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. * * . PREFACE 

This study was" conducted for the Office of Program Evaluation of the Department 
of Education. Pursuant to a competitive procurement, work began on thf study in 
the fall of 1980, most data were gathered during the fall and spring of the 
1981-81 school year, and analyses were performed and the report written 'during 
1982 and trie spring of 1983, In commissioning the study, the Office of Program 
Evaluation sought an analyse description arid impact oriented evaluation of the 
Indian Education Act 1 s Part A Program as implemented in public schools* The 
purposes-of the study were to assist Congress and administrators In 'the 
Department to manage the program and make decisions with respect to Its future.. 

From the study's inception, a. Technical Advisory Panel of Indian educators 
provided valuable support. They participated in the study's design, review of 
data col lection forms and procedures," and dita* analysis plans. Panel members 
were: Mlke'Charleston (Oklahoma City, OK), Teresa La Framboise (Lincoln, N£), 
Grayson No ley (University Park » PAL Edward Thomas (Ketchikan, AK), Joseph 
Trimble (Bell iwjham, WA), and Noah Woods (Maxton, NC). Each of the members of 
the panel devoted^ considerable time and attentlori^to the study, which was greatly 
appreciated* . ' v = : ' 

' . ■ ■ ■ ■ ■'. • ..' . 3 , ■ 

Development Associates also, acknowledges the valuable participation ^i the study 
of ^Sear Chief and 5 Associates of Missoula, Montana, B^ar Chief was an important 
subcontractor throughout the study and was involved in all of its phases* In * 
addition, several other^ Indian organ izations^and firms as well as scores of 
consulting specialists made significant contributions. Mike Charleston, Joseph 
Trimble, Grayson Noley,* Lloyd Elm and Teresa La Framboise ma,de particularly J 
important contributions In the analyses of data and preparation of this report, 
John Tippeconnic provided valuable ass1,stance^at various points throughout the 
study, as did others at the Center for Indian Education at Arizona State. 
University. The American Indian Institute at the. University of Oklahoma hosted 
the training of field data collection teams and made other significant * 
contributions in a number, of areas* the . involvement of each of these individuals 

and organizations was highly important. 

... 
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Throughout the study, the involvement of Dorothy Shuler, project off icer,.- was 
substantial, and she was a positive force at all times. At various times, others 
in the Department of Education, particularly Patsy Mathews, then of the Indian 
Education Program, and Keith Baker of the Department's planning staff, provided 
particularly useful advice and were otherwise quite helpful. Finally, we wish to 
acknowledge the literally thousands of Indian and Alaska Native students, parents 
and educators who participated in the study and the staff of the local school 
districts who were Involved. With almost no exceptions, these individuals were 
gracious and most helpful. . '•■ 
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"THE STUDY DESIGN AND METHODS USED FOR THE EVALUATION OF THE iMPACT OF THE, - 
PART A ENTITLEMENT PROGRAM FUNDED UNDER TITLE IV OF THE INDIAN EDUCATION ACT 

' CHAPTER It INTRODUCTION ' . , . . 

This monograph pnovi des a* "summary of the key evaluation Issues, design 
approaches, and statistical techniques used In conducting the Impact evaluation 
of the Part A Entitlement Program of the Indian Education Act, In 1972, the 
Indian Education Act was passed by Congress, and an Office of Indian Education 
was created, within the U.S. Office of Education. For the first time* a major 
Indian education program was located outside the Bureau of Indian Affairs <BIA). 
Al though there were difficulties In the initial implementation of the Act; the 
Office of Indian Education was abl e to provide funds to public school districts 
for the first time. In. 1973, to meet the "special educational needs of Indian 
students. 11 K * : 

The Part A Program provides formula grants, upon approylfid applications, to loca^ 
educational agencies provided they meet specified criteria relative to the 
enrollment of .Indian children within their districts. ^ To receive a grant, 
eligible school districts are to have establ ished an Indian parent committee; to 
have conducted a needs assessment to determine the spec fa 1 ; e duca t i o na T needs; for 
the Indian students enrolled in their schools; and to have developed a program 
plan based upon those identified needs. ; - 



A local education agency is entitled to receive a grant if tfie number of 
Indian children enrolled in thal^ agency's schools either totals ten or more, 
or represents at least half the total enrollment for that agency. However, 
an agency may a^Ty .without regard to the enrollment requirements if it is 
located in Alaska, California or Oklahoma, or Is on s or In proximity to, an 
Indian reservation. - * 
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The Part A projects are, on the average, small/ provide supplementary services to 
Indian/Native students^ and are quite diversified In terms of program emphasis 
and intensity of program activity; In the 1 981 -82 school year, the per-student 
expenditure by the Part A projects surveyed was about $221 ; they had a median 
budget of $26,450. * - ' ' ■ — ^ : 

' . ' " . . ' ■• ; ^ . , ..... . . ■ . . . . 

Federal regulations state that "projects may focus oh basic skills Instruction or 
other academic areas or* on Indian culture as related to academic skills . . * an4 
projects are encouraged to use cultural ly-based materlals/and techniques^ fn 
program activities," The permissible activities are quite broad, and include 
those, whlcri are not strictly academical ly^oriented but' which may be deemed 
Important activities for American Indian/Alaska Native students . > Exampl es 
include" home-school liaison services; traditional Indian arts, crafts, music, and 
dance; comparative cultural studies; school -related expenses; and native language 
arts. - , . - - - 

Many project activities are assumed to positively affect ♦students' self-esteem, 
pride, attitude toward school , and attendance. Thus, the Program may have had 
impact upon students in many areas in addition to academic Improvement; 

Project impacts upon participating students were not the only evaluation 
concern. Federal regulations also require the Involvement of parents in a parent 
committee. In fact, formation of a parent commi ttee is a prerequisite to Part A 
funding* In addition, the legislative history of the Indian Education Act 
reveals a concern with the educational context in which Indian/Native students / 
are expected to learn, Thus, other areas of potential Impact consist of effects 
upon the Involvement of parents in their children's education, as well as impacts 
upon the school districts, schools, and staff. 
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It was acknowledged at the onset that 4he major aval uation problem was the design 
a flexible evaluation approach that {leal t accurately and sensitively with . 
identifying and measuring past and current impact » within and across all 
dimensions of possible program emphasis, without misrepresenting the Program and 
its Importance, This was acknowledged as difficult because, as noted, projects 
are small , supplementary in nature, and quite diversified in "terms of program 
emphasis and intensity of program activity.^ 

Another major problem also existed- In spfte of an acknowledged need to assess 
the Impacts of the Part A Program after nine years of operation, such evaluations 
could well be viewed .in negative and threatening terms by those involved, thus 
affecting their full cooperation. Moreover, from the Indian perspective — *' 
because of the unique nature of the program — there was the percei vech danger 
that Inappropriate measures or standards might be f used In evaluating the ' + 

Program. Thus, the Program could be misrepresented and misjudged In its 
Importance, and Its Impacts upon Indian/Native students, the Indian community, 
and the schools coul d be misinterpreted. 

With such considerations in mind, a ^reat deal of time and effort was Invested in v 
developing an evaluation plan responsive^ enough to assess accurately and 
sensitively the myriad possible past and current Part A Program Impacts,' 
recognising all the while that the Program was not the only intervention likely 
to affect Indian students. Besides .regular school programs , Chapter 1 , Johnson 
O'Malley Act programs, and other federal effort*/ tribes and Indian organizations 
in many areas operate special educational programs focused on academic and 
cultural activities, Specifically, the evaluation was designed to: 

• Capture and describe in detail the programmatic and Individual project 
elements, as well as the characteristics of the school districts ? 
communities, and those individuals who may have been accepted by the 
program; and 

• Assess the extent to which program activities, both academic and 
non-academic* have contributed to discernible impacts over several * 
dimensions — human, programmatic, and organizational. 

i 

: -_ - . ^ _ , , , pi , ._ _ , T _ _^ r ^ ^^ ^ ~-~--~ ^■^■^^-^ D^Tm*OPSraNT ASSOCIATES, INC* 

ERIC 



Thus, this stu<ty may be conceived as a multl dimensional evaluation of the Part A 
Program, focused upon Identifying and explaining the" results of the_ Program since 
its Inception In 1972. It Includes an Inquiry Into ani analysis of : (a) the 
Program's origins and initial Intent; (b) the operations of the Office of Indian 

Education; and (c) local level operations, results ^ and ongoing needs based on 

■* * ' * ^ ■ . - * 

visits to a representative sample of local projects. . V 

'" : ' ; . * ~ ' 1 

Finally, several broad goals were outlined relative to the conduct of the overall 

study: " ./ , , . 

* * ' t ''" • ^ * * •* 

* ■ • To gain a thorough understanding of the Part A Program — its goals* 

purpose, history, characteristics, and operations at all levels, as well as 
its place in the education of American Indian and Alaska Native children; 

• To maintain and ensure sensitivity to, and involvement of, the Part A 
Program participants, as well a^the potential^ and eventual users of the 
stucty results; ."-"■"■« # ! ** w 

• • - 

• T^ Involve Indians in all aspects of the study, especially the initial 
design, instrument development and field testing, data collection, 

* analysis, and interpretation of the results ; 

• To develop a sound evaluation design responsive and flexible enough to; 

(1 ) Capture the diverse nature of the Part A Program; 

(2) Incorporate emerging concerns Into existing design, data collection, 
and analysis procedures; and 

(3) Adopt alternative approaches and strategies as required to mesh real 
wor*ld conditions with the conceptual evaluation model; and 

• To Insure a representative sample of projects in the study In all types of 
settings (urban, rural, reservation, near-reservation, and village) and 
individuals who are involved In and impacted by the Program, 
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To provide an overview of the overall Impact evaluation design? the study may be 
viewed as* being divided into four discrete stLrdies^for the purposes of ' 
information acquisition find data collection (see Table 2-1 X* They simi ^y , 

• : * * \. - -= , . . ; .\ ' ' ■ • ; - - ^ \- ; f • • : 

The Preliminary and Exffl oratory Study; ? " 

The Local Projects Impact Stuc#; ; ? ".: ; = ?.'-,; 
The Alternative Resources Study; and 

The Small Rmjec^ Study- ../ ;.-„.-/. / - s " s 

The Pre! i mi nary and Exploratory Study was essentially an evaluabi 11 ty assessment 
and Information acquisition endeavor, designed $o provide a comprehensive 
background of information relative to: 

• A review of literature pertinent -to Indian: education evaluation ? 
- specifically, and impact evaluation literature, in general ;- ' 

• A review of the Part A legislation^ hearings, and regulations; 

, • The development of a case history of the Indian Education Program — its 
activities, operations,, and characteristics; " . ■■ 

• A review of Part A docuni£nts; applications, and other local project and 
district material on file with this Department of Education; 

• Personal interviews with congressional* staff, and Department of Education 
and IEP official s and staff; and » 



• Visits to nine Part A^projects representing a diverse cross -section of 
settings, sizes, and types of program activities, 

% ' ~ ■ : - - . .■■ . ; . 

The results of these ^ctivit^s -provided the framework for the development of the 
conceptual impact evaluation design which, in turn, provided a gui de for the 
evaluation of potential Part A impacts. However, the diversity of expectations, 
needs, activities; and settings became clear during this early assessment* 
Except at the broadest of levels, there were substantive 'disagreements among key 
staff within the Department of Education and also among congressional, offices 
regarding the purpose, of the Part A Program and criteria by which it should be 
assessed. For some, the' Program was perceived primarily in terms of improving 
basic academic skills. For others, its primary purpose was to heighten awareness 
and understanding of Indian history apd culture, or to Increase student pride and 
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. ^ T ABLE- 2-1 
OVERVIEW OF IMPACT EVALUATION DESIGN 



Background Study 

- 0>- 


Exploratory 
Inipaet Liter 
legislation 
Review 

* 


Itudy^ 

anil Regulation 







Oli/Pederal Level Study. 
• (I) ^ 



Case History of 

on Activities, 

Operations* and 

Characteristics 

Document Review Of 
Oil and Local Material 

Personal Interviews with 
Congressional staffs 
USED and Oil officials 
and staff 



Local Prpjdct- 
lapact Study* 

(3) ■ 



Operations/ OuteeifcS 

***** 



Representative 
sample of 
refunded 
projects; SEAs; 
Indian Evaluation 
a Tecttnlcal 
Resource Centers 

Structured 
interviews, and 
docunent reviews 
(and supplemented 
as warranted by 
unstructured topic 
guides) 



Alternative Resources 

* Study 
: . (4)* 



Telephone Survey of 
; Unfunded LEAs 

Review of Federal v 



Matur e* -.Extent ♦ #hd Ii^rovements Note d In iapact Areas 



Student Oriented; ..- - - - 

Acad&nic Improvefiiehts 

School Attendance, Retentrf ™ and J^f* ,at ™ n fmM4 
Attitudes Toward School. Self and lelng An Indian 
pledge of Post Secondary iducatlenaK Opportunities 

Non-Student Oriented: ' v 

^Involvement of Indian Parents In a"?^ 1 * Etotfon - 
Involvement of Indian Parents 1n PuMic School OlstHct 
nm °tndleji-Miated GurHouluai Or TOtf MateHaJ 
SiS !Slcnoots and District Toward Indian Students 
Employment of, Indians Sy Schools - - 

Support of Indian COTajnl^ -for Puolic School Districts 



| Current Levels of Prograia ^ctlveness. Continuing Meeds for Service, and ^ec^ndatl 



ons for Future Action 



•Approximately 80-90* of the Evaluation fMogrws are a?* 0 ^ 6 ^ J*^* 
flfeudv 3) The purpose af studies 1 ana I are to provide Important contextual 
Md fyipreSve information relevant to the linpadts of local prajra. The 
ouppes?oflIudy 4 is to address several topics of particular interest td 
Congress and to provide Infarmtlen which «1 11 tm considertnfl tte 

continuing need for and directions of the Part A program, J. Q 



9 

ERIC 



Dfratorasif Associ^Tss, Ere. 



■ : '"'V' '"'- r 'T .■>''•. '"■ **" ' ' «. . '• ' • : 

. ' ■■ ' . . " . * ■. ' V . . ' . • . • • • ' 

. - V' = • v : • ■ / ' ' ' 

sel f -esteem. Also, for many (but not all) the primary. goals of the Program 
Included bringing about attitude and policy level change within local education \ v * 
agencies. Similarly*, many also considered increases in the involvement of Indian 
parents in the^ local school system a critically important dimension of .the / 
Program.; * * - ; ) / • ^ - 

These varying perspectives were held by persons with legitimate interests and 
influence In the Part A Program, kn'd had Seen communicated, at* least to some * 
. extent, as priorities to local project starts'., frhus, in planning the evaluation, 
it Was decided to gather evidence of and report^on Imparts on each of these * 
areas. However, It was recognised that because of limited project resources and 
the federal mandate thaVprojects be locally designed, it would be inappropriate 
to expect radst projects to have discernible impacts In all J ^ 

? The Local Projects Impact Study was the central focus of the evaluation, and its 
J5 the major focus of this report. In brief, the. data for ^ this study were 
collected from a stratified random sample of 115 of the Part A projects funded 
during the 1981-82 school year, which were funded for three consecutive school 
years (1979-80, 1980-81 , 1981 -82) and which reported having more than 30 Indian 
students in 1980. By screening out projects which had not been consistently 
funded and projects with small numbers of students, the study was assured a 
sample of projects with sufficient numbers .of students and with a history that 
would allow the gathering of longitudinal data which could be used to provide an 
assessment of project .impacts over a period of time, A total of 865, or 85 
percent of all Part A public school projects funded during 1981 -82, remained In 
the universe after screening (the others were included In the Small Projects 
Study or the Alternative Resources Study described below), Interview or 
questionnaire data were gathered from a variety of respondents from each of the 
ITS projects in the study sample. School records were also consulted for 
Information concerning students, project activities, and the characteristics of 
the school district itself. 1 



1 A description and the results of these preliminary activities are presented 
in: Reimer, J, Interim report on the Part A Program: A review of "findings to 
date, March 1981. . ' 
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In essence, the impact eval uati on wis a -major •effort that ftecessari ly^^jn^Tved % .; 
many tasks, ^unparized, they are: % f v / . \ ^ ^— .-].'■ i 5 

• Identifying the overall dimensions of the data requirements (i.eC, types of 
interviews, 4 questionnaires, document review forms *and formats. Individual 
Item and question development* and sel ection of Impact indicators ); 

• Developing, field testing, and modifying, of the structured data collection 
f nstruments ; * - v \ > _ ; w 



etc, , 



• Defining^survey populations of projects, s tude/its , ^Barents , teabAers , 
from 6 which samples would be selected; v- . ( * * 

> /. i ■■ . ; . * '. " r * ;* " <"_*"■■ - .* 

• Determining the sizes of various samples; V* k 

• Designing and selecting, the samples; *•"*; W ^ L * - 

• Establishing communication with project directors and staff ,vteachers, 
students , parents , and others as needei tQ perform data collection 
operations; v . ; r ; : : ^ 

• Determining data collection periods; 
. Identifying, selecting, and training field staff; V , v 

• Conducting field data collection; * ' ' ' . * 
» • Processing data and developing storage procedures^ / 

• Developing the analytic plans* and "* ' . 

• Developing plans for the unstructured qualitative aspect of the data 
collection* , 

Tables 2-2 and 2-3 provide a listing of the data collection instruments and the 
number of respondents or school districts from which data were obtained for each 
instrument, - ^ • 

The Alternative Resources Study examined the federal , state, and local funding 
resources, othir than funds from Part A of the Indian Education Act, for the 
education of Indian/Alaska Native, children, The results are based on a. 
probability sample of school districts with Indian students* including districts . 
'which do and do riot receive Part A funds. Projects which were not visited as 
part of the Impact Study were contacted by t ^ephone, and a review of pertinent 
federal program materials was also conducted. , 

,„■ . ■ . . ' 20 .... 
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K TABLE 2-2 • 

RESPONDENTS TO FALL AND SPRING! INTERVIEWS AND 



QUESTIONNAIRE ADMINISTRATIONS 



/ 



.J 



Respondents 



District administrators* 
Part A project directors- 
Part A staff 

Tutors of Indian students V 
Indian students, grades 4-'6** 
Indian- students, grades 7-12** ■ 
part A#a rent committee chairpersons 
Part A parent committee members 
£ Parents of Indian-students 4 
Elementary and secondary school principals 
Regular classroom teachers of Indian students 
Indian community /tribal leaders not associated with Part A 



■ ■ - ": 


Fall . 


Spring 


<p * 


• i 

154 


' 109. 




114 


108 




.'■ "4 * 


413 






329 




" 5,393 


5,201 




8,145. . 


7,369 




114 








290 



1,546 
.450 
1,307 
102 



■*Titi=superi 
Spri ng=Supervi sors only, 



all=Superintendents and supervisors of 1 Part Aproject directors, 1 ' 

ted 



**A discussion of the representativeness of the student sample is presented in 
•Chapter 3, Sampl ing PI an see ti on , part 3 (f ). • / \ 



TABLE 2-3 * 
DATA COLLECTED FROM SCHOOL RECORDS 



Type of Data Collected 


. ; . 

Number of Cases* 


School district characteristics 


• 

. ' '116 districts.'' , « 


School attendance of Indian students 
(5 years per student, where available) 


8,376 students 


Reading and math achievement test scores 
(spring 1981) ¥ - / 


- , 6,425 students (' ' ' 


Characteristics #nd progress of tutored 
Indian students** 


- m 
' 3,123 students 


High school completion data on. 5 cohorts 
of Indian sophomores (1971*1979) ■ 


• 

2,098 former students 


*Sehool record data were collected only once, 
depending on the type of data. 


either during the fall or spring, , 


^Tutors completed forms for each tutored student, 
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Finally* thf Small Projects Study Involved the examination of Part A projects 
receiving $5,000 -.or less In 1981 and/or serving 30 of fewer Indian students. All 
32 of these projects were selected, and, a review of their application materials 
was conducted at , the^Indian Education Program Office in the Department of 
Education. / ^ v - - . 

Telephone interviews .were attempted with project directors, parent committee * ^ 
tfiairs, and a district adnlnistrator at each project. Various materials were 
also requested and later reviewed. Three of the -most promising projects were 
visited and more detailed data gathered. - 

The results of the Alternative Resources "and the Small .Projects studies, arlong 1 
with the data* collected during the Local Projects Impact -Stucjy, provide* * / 

• A balanced and detailed picture of the current levels of Part A Program 
Involvement and effectiveness at the loca1^|evel ; 

• An estimate of the continuing need for federal assistance In the education 
of Indian children; and 

*± • The necessary basis upon which to? make recommendations for future action 

with regard to Part A and 'federal Involvement In the education of Indian - 
chil dren. 

The evaluation required a variety of descriptive and Impact data about the 
orojects, the people directly or indirectly involved, * and the settings (e.g., 
community and school districts) that surround them. Building upon the knowledge 
and information acquired during the Preliminary and Exploratory Study, an 
evaluation m design was developed which evolved as the study progressed, 
Indicators of potential impact were sought and assessed from many different 
sources and over several different dimensions, as noted In Table 2-4 (Analytic f 
Framework), These multiple impact indicators from multiple sources were used to: 

• Detect any form of Impact on that dimension, 1 f "such had occurred; 

: • Obtain a consensus or agreement among findings from dlffereltt sources that 
an Impact had occurred; and - 

, § Obtain a fuller and more balanced, picture of the types of Impacts that might 
be attributable to Part A, so the extent of Part A effectiveness could be 
better understood and the findings more easily interpreted to various 

'audiences* 22 

' ■* -_-..:>_-_ UM . — ^ _ ^ _ , . — development associates. irr< 



Broadly speaking, the foil owing six basic types of information were collected 
within the scope of a quasi -experimental Impact evaluation: 

• Descriptive data concerning project operations and' funding; * 

• Mediating variables that "explained" impact measures; . . 

• Confounding variables (I.e. f factors at the core of Part' A operations, such 
as funding from other sources, which make it potentially difficult to 
attribute outcomes to Part A); 

• Impacts on students; - . " ■;' 

• Impacts on parents and Indian communitiis; and\ 

• Impacts on school districts. 

The descriptive data were used to present a profile of the: \ 

• Students and communities served; * 1 

• Sources of project funding; ■ ' ' 

• Typical program components and services; 

• Project staff characteristics; and . 

• LEAs, schools, and so forth* ; t . 

Various descriptive statistics, such as means, -medians* and percentages, were 
presented* Graphic techniques such as trend lines were used where helpful . 
Cross-tabulations were us6d ^to Indicate characteristic patterns for project 
types, student grade-level ranges, and other breakdowns of special interest. 
TMs information provide^ a thorough understanding o^ the data available for 
Impact analysis purposes, • 

To that end, a series of balanced and comprehensive Impact analyses was 
conducted. The project, as well as the student, parent, and other data sources, 
were used as units of analysis, A variety of techniques was used for Impact 
analysis purposes. Trend analysis techniques (including plotting of group means) 
were used to detect shifts over time and/or between groups in various behavioral 
and attitudlnal project-level Impact measures. Such results Indicated whether 
project operations had positively affected the rate and extent of Impacts 
gathered from the district, parents, teachers, tribal leaders, etc, Factorial 



analysis of variance, multiple linear regression, general linear models* and 

multivariate analytic techniques were, used to determine whether ^.shifts 1n 

student-level Impact measures (e.g. , attendance, self-esteem, cultural knowledge," 
f * , / ^ ¥ ... - 

a'ttitudes, and overall progress t n school) Indicated a relationship between «. • 

student participation In Part A activities and ensuing Impacts, These analyses 

were conducted In a way that mak^s it possible to ascertain the types of projects 

in which Impacts were the greatest.' The statistical significance of the findings 

and whether or not their size represented tfieafHngful or practical differences^ 

were used In interpreting the^ results. As mentioned earlier factors that 

represented alternative explanations for the results were also controlled for or 

taken into account when conducting the Impact analyses, , / ; 

Extensive quality control procedures were built into the data collection plan, 
including: . • 



• Formal, training of I field staff; 

• Specific staff responsibilities titv site; 

• Monitoring data collection rrftes of all types * 

• Coordination of field staff and team leaders with the corftractor 8 s central 
office supervisory personnel r and - 

• Procedures for dealing. with contingencies experienced on site, 

Most analyses were performed on a data bqse that was designed to permit both file 
(i.e., particular respondent) and subfile (i.e. , subgroups of respondents) 
analyses. This was done because certain analyses were better prepared at the ^ 
project level, and other analyses at the student, parent, tutor, or other 
individual respondent level. 

The structure of the qualitative and quantitative analysis procedures are quite 
different, Jhe qualitative or naturalistic approach in its pure form does not 
provide a structure for the analysis of the "raw information" collected. The 
information is not necessarily collected In a manner which allows consistent ^ 
aggregation for systematic*, statistical analysis procedures. Nonetheless, 
systematic aggregation is possible and desirable at some levels, especially since 
the information gathered may be rich in detail and locally specific in content, 

■ 24 
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The "analysis process" used in this study therefore involved, searching few 
themes, consistencies, inconsistencies, diKect personal experiences, portrayals* 
faithful representations', ete,^ This was .a process of winnowing out Issues t and 

questions' The analysis and compilation of the results was not necessarily a 

■ \ . . ■ : f . * : . * • '■ 

■different and discrete operation or activity, different for 'each approach. 

- / •* -V * i • • • . . • * '•■ ". : 

Viewed as being complementary to each other, both approaches and their results 

. . * ■ \ ■ t - . ■ .' . ■- • 

were combined to fill gaps in the Information/data base (although information and 
data were collected irr different fashions). Specifically^ the Topical 
Qualitative Studies were reported fori the most part in conjunction' .with the * 
quantitative survey results, * ' > 

Finally; impact question^ addressed by , topical studies were subgrouped where 
possible under broader research, questions. ^Results for topical studies were 
synthesized both for the narrow impact questions addressed', as well as for the 
broader questions. Where topical studies .addressed identical or related impact 
questions (as did the survey instruments), the. results- of both were 
compared/synthesized/ Similarly, where qualitative studies incidentally picked 
up Information relevant to results of the %yrvey, this information was 
Incorporated in presenting the quantitative results of the research. '. 

Organization and Context of the Findings 

From the outset, the Department of Education made it clear that this study was of 
concern to a variety of quite different audiences. Various groups witfiin the 
Department, the Congress, the Native American Community,, the state education ^ 
agencies, and the 1 ocal* school districts have an Interest In particular findings 
-and the results overall, Early in the study, It became clear that not all 
potentially relevant Issues or questions coyld be addressed within the existing 
resource limits. It was also clear that the various audiences would not always 
agree in thei r interpretations of the Part A Program's historical context, study 
findings, or the preferred style and format for presenting results. 

Consequently , various strategic decisions and accommodations have been made 
throughout. With respect to analytic contents the study team attempted to 
address the key Issues raised early in the study by congressional staff, 



responsible persons in the Department of Education-, and members of the Indian 
community on the study advTsory. panel.: Regarding; the context of findings," V , 
middle ground has-been sought. The final report does not purport to thoroughly 
address the long and complex hi story of Indian education in the United States. 
Similarly, a middle grouhd was sought vrith respect to" Interpretation and- : 
presentation of results. -Where deemed apprbpri ate, interpretations of ■findings 
are offered and extended analyses were made. Elsewhere, however, the data are 
largely allowed to "speak *or" themselves;; as their meaning is embedded In 
controversies over values or interpretations °f ^•^1>^"Mi^nC*li ; fi^ti*: 

" Presentational ly, the object has been to provide sufficient detail to permit a 
fai r and complete assessment of the study 1 s-f 1 ndi ngs and of the Part A Program 

.without overwhelming the reader with a mass of specifics or technical terms. 
• • * - . .,*■ ■ •••* 

Indian Involvement in the Study ' • '• ' 

. i -=:••-• - ... 

. „ * . ■ • . " i " ----- 

The Department of "Education required that the final design and implementation of 
the study make heavy use of Indian 'professional s. Thus, from its start, the ; 
study was designed to provide for the substantive involvement of Indian educators 
at all points. For example, as" much cultural matching of interviewers with 
respondents as possible was required in order to increase the probability of 
gaining valid and useful data. • Similarly, it was required that Indian 
researchers play a substantive role in the study's conceptual design, the 
devel opment $f daita , col 1 ecti on 1 ns truments , and the data analysi s arid " .> - 

interpretation. * • ' ' 

- . • ■ .■ • • ■ .. ■ 

• * - ' .... 

Consequently, during the early stages of the evaluation, Development Associates 
identified and initiated ongoing consultations with knowledgeable Indian and 
Alaska Native educators who possessed a variety of experience and expertise. 
These individuals were known to have a thorough knowledge and understanding of 
Indian communities* organizations, and education in general, and of the Part A 
Program In particular." These consultations and Involvements provided insights 
and strategies bearing upon the" study's design. Implementation, and analysis. 
They were also quite helpful in establishing the study's cradi bill ty within the 
Indian and Native communities and in securing cooperation from schools, both of 
which facilitated data collection throughout the study." 



In addition, a technical adviso ry panel to the study was constituted* This panel 
was made up of Indian educators, and researchers from across the country who met 
to review and advise; the study 1 -S staff with respect to stilly design, data 
collection, and analyses. Thejr participated in decisions regarding areas' and 
strategies of Investigation; reviewed all instrumentation ; pirtid pa ted i n 
sampling and dita collection de^oision*making; reviewed pNltmfnary^Aa and 
advised regarding the analytic ^lans; and Individually reviewed and commented on 
drafts of thi s report* Through «ut the study, al 1 of these individuals devoted 
significant time and energy^ to ~fche effort* Consistently s they Were free in their 
criticisms of draft materials awid made valuable contributions to the evaluation 
overall. 

~ • .... . ~ • • . . ; •;=. 

The study also utilized over 60 Indian professionals as paid consul tants at ■ -. / 
various critical points. Throu g hout. Bear Chief and Associates (ah Indian owned 
and staffed consulting firm) served as a subcontractor to Development Associated 
During the instrument developme^nt stage, some 40 Indian educators — teachers,; 
project directors, and university professors — reviewed and offered revisions to 
the various questionnaires. Da^fca collection itself was undertaken virtual ly 
entirely by Indian or Alaska Na~*1ve professionals. Field staff were provided an 
intense week of special training prior to the fall data collection and a second 
week of training prior to the pr—oject visits in the spring, The field staff 
consisted of university faculty members, upper level graduate students with prior 
work experience, and experiencec^ professional staff from Indian flpfis or 
organizations with whom subcontr — acting arrangements had been made7 



Data analysis also Included heaw^y Indian Involvement. Native American graduate 
students at Arizona State University conducted the coding of all open-ended 

jnnalre responses from Native American interviewees. It was clear during 
'the earljy analytic phase of the study that Indian educators could more accurately 
analyze/ the responses of Indian parents and project staff than could coders with, 
If ttle icultura! or programmatic Insight* Similarly, Native American graduate 
students from Pennsylvania Stata University were used to provide ratings based on 
a qualjtQrtt¥# assessment of recorded data of the cultural components of the Part 
A projects* More substantively^ several Indian researchers were directly 
Involved In planning and conducting statistical data analyses and writing ^raft 
sections of the study's reports. 
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CooperatTon with Local Education Agencies 

The evaluation! design required considerable time-and .cooperation from the local : 
school districts that became Involved* In addition to Part A; Program staff and 
members of the Indian community, superintendents, assistant superintendents, 
principal s ^ and teachers were surveyed and files reviewed. Frequently, 
developing the sampling frames for the selection of Indian students and securing = 
school attendance records and achievement test scores oh Individual students 
involved considerable time from school personnel, 2 With few exceptions /school , 
districts readily agreed to participate 1 ri the study and maintained their 
cooperate on throughout the school year, 3 Typically, school personnel were " • 
Interested in their Part A projects and desired to assist in the Program 1 s 

evaluation* ■ ■■- — ■■-■■-A " - s - * - - v • •- ■...^.,-.-.>..^.^ v :.v- ■- 

. . . ■ ' ' . ' ■ - . - V . .. •; ; : .. r \ ■ ■ ■". ; ■ . .; \.:;yr- 

Local cooperation of school officials was encouraged somewhat by offering to 
provide local staff with feedback on their projects. To accomplish this, the 
study design called for supplying project staff with Information which could be 
useful in planning future activities collected from their sites as soon after the 
spring data collection as possible. Thus, during the fall and early winter of 
the 1982—83 school year, the participating projects were sent summlri zed results 
of surveys of regular classroom teachers, Indian students, and Indian parents at 
theirs si ties* Also, most projects were provided with analyses of Indian student 
achievement test scores and with school attendance data comparing Indian, students 

in their districts with state or national norms. 

" ■ ■ . ■ ■ ■ 

- I , . _ ■ 
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2ln some cases, the cost of their time was reimbursed through the study, but 
often it was not. 

3jhe extent of project replacement and attrition is discussed in Chapter 3, 
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CHAPTER 3; STUDY METHOI^ 



Major Features * , , • . 

1. Overall Purpose , 

"■ — ' ". • > ., < 

The overall purpose of this evaluation was t</ reach some conclusions about the 
effectiveness of the Part A Entitlement Program. Thus,- the broad study Issue 

was defined as follows: ." - 

• , - _ - . .... ■ ■ . i 

To what degree does the Indian Education Act Part A Entitlement Program 
- contribute positively to meeting the special educational and culturally 
related academic needs of American Indian/Alaska Native children and youth? 

2. Objectives ? t 

Three basic objectives were drawn from this broad study Issue: 

• To describe accurately "and sensitively the range of Part A project 
objectives, target groups, cultural activities, and program funding 
patterns; 

• To determine the nature and extent of Part A Program impacts on 

Indian/Native students, their parents, and local school districts; and 

- * - 

• To determine what s if any, changes In legislation or regulations will 
provide a more systematic and effective approach to meeting the 
educational needs of American Indian/Alaska Native children. 

3. Study Questions - 

To address these objectives, a series of major stuc(y questions was developed, 
and is shown In Table 3-1. These questions guided the identification of 
Information sources and types of data required and aided in the selection of 
analytical tools. * 
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' TABLE 3-1 • . ..'/•'■• r 

•MAJOR STUDY QUESTIONS GUIDING THE EVALUATION ; ' 





1. 


What are the organizational, fiscal, and human resources avail able to* 
Part A projects, and how do projects utilize these resources? 




2. 


to what extent do the objectives of projects funded under the Part A 
Entitlement Program address the special educational and/or culturally 
related academic needs of American Indian/Alaska Native children? 


- 


3. 


How have Part A project activities been implemented? 




4. 


What are the impacts of Part A projects on American Indian/Alaska 
Native students? .. 




5. 


What Impacts- do Part A projects have oh the parents of American 
Indian/Alaska Native children and on the American Indian/Alaska Native 
community that projects serve? 


f 


'6. 


What impacts do Part A projects have upon their liEAs? 




7, 


How do federal*leve1 activities, especially those of the Office of 
Indian Education, affect Par^^projects? 




8, 


What is the total amount of federal education funds expended by local 
school districts on Indian students in grades K-12, and how many of 

these students are receiving various types of special services? 

1 ■* 

* ** ' - - 
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Several of the studly questions (1 , 2* 3, and 8) were primarily descriptive in 
r nature an%tiius addressed tire f i rst objecti ve. ; Other questions (4* 5, and 6) 
focused primarily on the Impact of Part A, uie third objective was also 
addressed through question 7, which analyzed Indian Education Program's 
policies and activities for strengths and weaknesses. A set of related study 
questions was also developed; It is contained In Appendix 1 . 

4. Research Approaches | - 

Two basic research approaches were used during the evaluation of the Part A 
Program. They are the quantitative and the quail tative/fiaturalistic 
approaches. The quantitative approach involved collecting data on a 
representative sample of projects, using structured data collection 
instruments developed during the design phase of the study. Qualitative. 
Inquiry was conducted at the same time In order to examine a number of 
specific issues/topics in a subsample. The v purpgse of blending the two 
approaches was to achieve a broader and more balanced Information and data 
base and to provide a fuller, more complete picture of the Part A projects 
from which to derive results and conclusions. What follows is * summary of 
"the two approaches. 

With respect to evaluation, the quantitative approach was designed to raise 
and analyze certain kinds of questions and Issues regarding a set of 
previously identified potential 'areas of impact. This approach relied on a 
structured ^set of procedures in an evaluation design whose elements are 
delineated prior to actual visits to the projects* (Hypotheses ^tested are 
derived a priori . ) The procedure made use of structured forms and schedules, 
such as questionnaires and interviews addressed to particular Individuals, 
groups, and data sources- The objective of such an approach was to 
systematize and categorize the data collected Into a structured form for ease 
of analysis, using standardized descriptive and inferential statistical 
procedures. The procedure essentially sought to discover variables and 
describe their inter-relationships, primarily for purposes of prediction and 
control. Being reductionist, the process attempted to converge on the truth, 



Quantitative data were collected in the form of structured interviews and^ 
sel f-adnrfnistered questionnaires and from Records and profiles. (Refer to . 
Table 4-1 for, a listing of the structured data collection forms and number of 
respondents to each. ) V ;.. ,V - 

The qualitative inquiry, on the other hand, represented an unstructured 
approach utilizing case studies as well as ethnographic, naturalistic ■ > 
coneepts, as a way of adding, more breadth and depth to the overall evaluation 
in a fashion that the structured quantitative approach is not Intended to do. 
(The topics on which the qualitative Inquiry focused are presented in Table 
3-2.) \ ■ ■ " ; V.v- 



*E 3-2 

TOPICS INCLUDED IN THE QUALITATIVE PHASE OF THE EVALUATION STUDY 



1 , Effect of project activi ties on students going to col lege or other academic 
programs after high school (using anecdotes , special cases; rare events); 

2. Effects of changes in staff at the National Office of Indian Education on 
local project operations; '\ . . ■ ,. t[ 



3. Need for (additional) technical assistance by projects in curriculum design 
and developing other program materials; \ 

4. Impact of Part A projects upon the academic achievement of the Indian 
students, presently or in the past; and 

5. Impact of local Part A projects upon the attendance of Indian students in the 
school district, presently or in the past, 
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This approach was also designed to raise and then analyze certain questions 
and issues regarding the impacts and results of the Part A Program* The 
approach was essentially open-textured and evolves as tjie evaluation 
progresses; thus, it was open to identifying and exploring any range of 
Impacts pr results if they are present. Since It was not limited or 
constrained by strict desfgn considerations, questions were formulated and 
answers were sought that are specific to the immediate situation. Since this 
method could be described as focusfng upon multiple realities, it was 
expansionist, and phenomena did not converge upon a single truth, but diverged 

Into multiple truths. — . * 

" 1 ■ 1 ■ ■ -, - — ^— — ^ , M?KiOFMENT ASSOCIATES, INC 



'• '•"« " ,;"/ . :/ r ' V , ; 

great weaHh and depth of Information cowl d be gathered in this qualitative 
fashion, and the approach has the flexibility to examine alj possible Impacts 
as they were Identlflek as well as to fill in information gaps left by the 
quantitative study. The richness of the information gathered was also a 
strong point % in this process, provided the information is blended with the 
information gathered from the quantitative approach** * 

Jtt each site, qualitative data ^relating to the research question (s) selected 
for that site were gathered, analyzed, and written up -by the field staffs at 
that site. A single study staff member acted as editor/analyst for each of 
the topical studies from all' sites, being responsible for going over eadfi 
study and checking weak points with authors, and performing Internal synthesis 
of the qualitative studies with results from quantitative survey data analysts. 

Special Design Issues - 

* m 

1. Design Constraints 



Three major factors Influenced the design. First, certain of the 
legislatively-defined program outputs are not easily measured. There are no 
generally accepted definitions df the legislative goal of meeting the special 
educational and culturally related needs of .Indian students, nor are there 
accepted definitions or measures of attainment of this goal. Second, each of 
the school districts has broad discretion in the Interpretation of the Act and 
the use of federal funds* This variably ty required extensive onsite data 
collection and the tailoring of the overall design to local contexts and 
objectives. Third, the nature of the projects and the school -community 
systems In which they operate made attribution of impact findings to the Part 
A Program difficult. 

* 

Almost all eligible LEAs and most Indian students within these* LEAs were 
involved to some extent In the Program, thus virtually eliminating the 
possibility of "true experimental control groups" of any kind. This 
situation, combined with the supplemental nature and relatively small size of 
the projects, and the fact that most projects have been Jn operation for more 1 
than five years, compounded the problems associated with Identifying student \ 
and Institutional ctfkngJs attributable to project activities. 
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As a result, extensive discussions about program expectations and operations ; ^ 
were undertaken with relevant congressional staff, policy and operational 
Staff In the Department of Education, national Indian organizations, and staff 
of local school districts participating in the Part A Program- Resolution of _ 
problematic areas was approached in the following manner, ' 

. . . " . ^ - •.: • . 

2, Approaches Adopted 

To accommodate the diversity of settings, activities, and 'objectives, a core 
set of activities and Impacts was Investigated at all sites, with other 
' activities and Impacts explored only at sites where they were emphasized. . 
Data collection forms and procedures; Included the identification of relevant 
project components and activities and use of certain instruments or sets of 
questions In sites where they applied. On another plane, various analyses 
dealt with this diversity by using data from sites where certain objectives or 
activities were absent, as comparison sites for those where they were 
present. In addition, by scheduling data .collection Visits In, both tJiefall 
(1.981) and the spring (1982), it was possible not only to measure such Impacts 
within a school year, but also to utilize data gathered in the fall to refine 
on a s1te-by-site basis spring data collection plans and procedures. Analyses 
-of data collected In the fall permitted more precision in Identifying levels 
and types of student project participation, and In developing procedures for 
gathering limited qualitative data at each site. This aided in data 
Interpretation and in preparing focused site descriptions illuminating Issues 
of special policy Importance [e.g., effectiveness of very small grants. 
Integration of services with Johnson O'Malley (JQM), Chapter 1, etc. 3. 

To contend with the issue of attribution, the study team and Indian education 
consultants built into the evaluation design a number of features that 
substantially heightened the likelihood of detecting impact. 1 First, the 



^ee Appendix 2 for the names and professional affiliations of the study s 
Technical Advisory Panel and other consultants who have furnished aid. In 
particular, the foil owfng have been extremely helpful in this effort: Mike • 
Charleston, formerly of the Native American .Research Institute, Morris Rosenberg 
of the University of Maryland, Srayson Nole/ of Pennsylvania State University, 
Joseph Trimble of Western Washington State University,, and William Theimer of 
the University of the Pacific. / 34 

' ' - - , — DETBXiOFSONT ASSOCIATES* IE 



design recognized that different types of Impacts justified the use of 
different assessnient**pproaches and techniques. Thus, a breadth of data from 
a variety of sources. and perspectives was collected, By doing so, findings 
could be "triangulated 11 as a basis for conclusions regarding, project impacts 
and effectiveness. Such analyses were strengthened by considering the local 
expectations and needs assessments of projects, regarding what impacts they 
set out to accomplish. 

Other related evaluation design features dealt with the issue of detecting 
changes In levels of Impact measures over some period of prior project 
operation. The design therefore used a longitudinal data collection approach, 
where feasible, to examine the extent' of improvement or positive trend of 
project performance. In such instances, a particular project acted as. its own 
baseline and basis for comparison. This feature was applied to a number of 
project Impact measures, and, as warranted, to student data such as attendance 
rates* In these instances, sufficient prior years were used so that an Impact 
trend Hni was present* For certain impact measures, the trend line 
comparison represented the time period of district operations prior to Part A 
funding, so that the time period became the comparison baseline for subsequent 
Part A funded years* The concept of having an earlier period as a baseline 
for later ones also extended to detecting improvements In student attitudes 
toward school, group Identification* and sel f -concept. To measure those 
Impacts, both the start and end of the school year were used, A series of 
cross-sectional and longitudinal comparisons by <jrade level, type, and extent 
of program participation (among other factors) was used in attempting to 
directly attribute Impact levels and changes to Part A* 

Other impacts dealing with the current level of effectiveness were analyzed in 
the light of what Part A project characteristics and activities were 
associated with these findings, This form of comparison (also termed/ 
"differential impacts 11 ) uses projects having certain features or students 
exposed to certain treatments as a baseline for projects or students with 
other characteristics, It was used an as approach because knowing the 
relative effectiveness of project efforts and settings is likely to be 
important to policy planners and Indian educators. Still other analyses. 



albeit a limited number, were conducted by using data from eligible yet 
unfunded districts (the Alternative Resources Study) to determine the patterns 
of funding and services they provide for Indian/Native students. Such 
comparisons helped to Illuminate that which Part A uniquely contributes. 

Areas of Project Impacts ' 

From these discussions It became clear that. the Program was expected to have 

Impacts In three areas: ■ s . • • 

" - - - - - fc 

(a) Student Impacts, The ultimate purpose of the legislation is to ensure 
that educational neefe^st Indian/Native students are met. Indicators that 
this has occurred Include Improvements in the following areas: 

■ „ • Performance In basic academic skills; J 

• School attendance* ■ * - 

• Knowledge of Indian (tribal) heritage, history, and culturally related 
topics; ' / 

• Involvement and Interest In school; 

• Student self-esteem; and * 

• Academically-related performance after leaving high school (securing a 
high school equivalency degree, attending college, etc.). 

There w^s also jyide agreement that Improvements An student attitudes 
toward school and toward themselves as Indians may be necessary to achieve 
these objectives, and thus are valid and Important objectives In 
themselves. This aspect Is particularly Important In urban areas where 
^relocated Indians/Natives and their children often Are disconnected from 
their cultures and tribes. Given the size and scope of most local 
projects, however, there (wis also agreement that It would be inappropriate; 
to expect Impacts in air areas in most projects or to admi nf%t,er 
standardized achievement tests In project sites* , (See In Chapter 4 the 
section "Data Analytic Techniques Used. 11 ) Thus, the study was designed to 
accommodate individual project settings and objectives, and to utilize 
measures sensitive to highly varied local Interventions, 



(b^ Parent Impacts, Congressional testimony leading to enactment: of the Part ^ 
A Program highlighted the overwhelming evidence of the alienation of / 
Indian parents from the public schools arid a widespread lack of parental : 
support for their students 1 school performance. For "this reason, the 
1 eg1 si atl on and subsequent regul ati ons have stressed parental invol vement 
In the planning and Imp! Mentation of local projects. Indeed, the 
congressional mandate for parental involvement 1s more explicit and far - 
reaching here than in any other educational legislation. The Program has 
also encouraged Indian/Native parents to be more active in their 
children's schools and educational activities. Thus, in discussion with * 
Individuals at the national and local levels, there was complete consensus 
on the Importance of assessing the Impact of the Part A Program on parents 
: and their involvement with the project and the school s. Parental impact 
areas of particular importance are: A * _._ 

•Involvement in school advisory and decision-making bodies (PTAs, 
project committees, school boards, etc . ) ; 

• General attitude and support for local schools; 

• New, non-school -rel ated involvement In community affairs; and 

• Personal gains In knowledge, education level ,"or employment as an 
outgrowth of project Involvement* 

. . . ' ; . J . .. , • 

(c)— School District Impacts. Legislative testimony and rationale for the Part 
A Program also Included strong evidence of the insenslti vity of most local 
school districts to the special educational and culturally related needs 
of Indian/Native students. Indeed, this need was widely cited as the V 
basic rationale for the Program and its special student and parent related 
activities and objectives* Thus, although few local projects Include 
Impacts ©n their schopl districts among the explicit project objectives, 
from a national perspective thir ls a critically Important area of 
Investigation which speaks directly to program continuation and : 
direction. From the review of relevant literature and discussions at the 
national and local levels, the school district impacts considered of 
particular importance are: 

; \ 37 . 
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• Use of curriculum and classroom materials that recognize the positive 
contributions of Ind1 ans/MJa ti ves to American life; 

• Employment of Indian professional personnel; . 

t Sensitivity of 1ns|ruct1orrnal" and bull ding-level school staff to Indian 
customs and values as thej**' affect students in school j 

• General school climate tovward Indian students and parents as perceived 
by the Indian community; asnd - 

.-■ • - ' . . • , • - • . 

• Modification of any preexisting school policies Judged oneroid by the* 
Indian community* ^ 

(d) Community Impacts , In additfVon to these i mpact^ , the Program may wel 1 
have had other Important, although unintended, consequences. These are 
particularly likely with respziect to the relationship between the Indian* 
community at largt and the piublic school systems, Consequently, the stuc(y 
design and instruments incorpoorated some questions that explicitly 

- 4 t 

4 addressed this potential Impa&ct area. 



4* Achievement Assessment 

- S ' ; : . - ■ ■ * " - : ; " 

In developing the evaluation studzty design, considerable attention was g,1ven to 

^^the appropriate use of standardised achievement test scores. It was initially 

presumed €hat these would be a cesnterpiece of the data collection and 

assessment effort* However, aftesr a thorough literature review, consultations 

with leading national experts on student achievement testing and evaluation 

' design, 2 a close look at Part A poroject applications, and exploratory visits 




2 Among others: Gene Glass of the Uni 3 verslty of Colorado, Ronald Berk of Johns 
Hopkins , Rodney Skager of UCLA* Wfl 11am Coffman of the University of Iowa, and 
Donald Ross Green of the California^ Testing Bureau. 



■ "- _ = / » - ' ? 

* ■ . f - . V* ' • • 

\ "■ . . \ ; - " ■ -\ ^ ' • ■■ < 

to several representative local projects* the Iflltf al expectations changed, 

Essentially, it was concluded that* the academic "treatments 11 prov*-i ded by roost \ 

projects were not amenable to assessment with standardized tests of basic 

acatdemic skills. Treatments were f recently either— too specific; C^e.g.V , 

tufcorlng in a particular sub tppt c,*f a particular class), or too di ffuse (e.g., 

Including some language arts instruction as part* qf^ an Indian student 'club's 1 

activities) to be captured by broadly-focused standardized tests. Also, most 

offcen, they were focused oh social studies, science, or other academic 

sut>-jects not addressed by such tests* These program factors pfysrr (a) a 

concern about the variety of unknown* biases in st$n dardized tests with respect 

to Indian populations; (b) reservations regarding -the statistical 

rel lability of test score data taken from relativ^l^ small nufflbtr« of students 

In any given project, grade level , and school ; and (c) a concern f^or 

com t-effectiveness and respondent burden created by extensfye; stc)(Sent : testing 

lecfcl to rejecting the use of achievement test s^ore *-clata as a priifery aspect of 

the impact evaluation design. * : . * 

, Ins 1:ead t it was decided to collect from all projects existing reding and math 
tes~t data for the spring preceding -the site visit (^3 .e,, spring ) 4 » 
Tes ~^t score data were collected from school files, converted to a c ommdh 
metwlc, and then cross-sectional ly compared aerois grade levels $tnd other 
stu~<ient- and project-level factors (controlling for differences in participant 
non participant student characteristics), In those projects with 



3lo da'Ei, no systematic study of item bias has been conducted among lend! an 
^students, to the best of the Writers 1 knowledge* 

* • • -\ 

^Data i5For only the previous year, were sought because pf prior expert e^c=e in 
seek img multiyear data from school files* year^to-y^fcr chartges In Atm strict 
testing policies, alternate grade testing, and tfte ggraeral condition of many 
school- files made it Infeaslble to seek rltrospictiv^l y mul tlyetf da^a in a 
national sample of LEAs. , 



• 39 



m= pEVEi.ojwrz' associates, Inc. 



0 

ERIC 



-28- 



students in tutoring components, pre- and pQ$t«ra;tf hjsag 0t» several academic and 
: Othfr school -related performance dimension! wre obt*t"^iti^ ( For those 
students, plus those not receiving an* parti cyir aw^^fti»i|cHa^rat, the^^^^^^^^ ; 
assessment of academic achievement was based on compos slt^ 'tern rat* ngs of 
teachers, parents , students, and staff. Using thes^ s^bpv-«:hes provided a : 
more- valid, locally-relevant, and cost-effective set »<iif Finnic ae=hieveirerit 
Information for Impact evaluation purposes thin tJcW^lv^Ilrect tsstlng of J 
students could possibly provide, Js ,.; ; 

5. Attitude Assessment ■ ■ , _ 

Particular attention during the design process was ^ to ttrie problems ^ 

of assessing Indian student attitudes and se]N$te^% Usllonally — recogniitd 
experts In this field contributed ext%ni1v#iyboth %Q ^fi^lssic evaluation 
design * decisions and to the development of stody fnst^rRji%ntSi 5 Aft£:er 

y extended deliberation and the Involvement of IndiaA dii^iofs across the 
country, It was concluded that data should trt coll #0^^ jpffl studer^ts .about ^ 
their attitudes toward themselves/ therir attitytes/to^^f^sctiool , and thiir 

K Identification as Indians or Alaska Natlvef* Ihasa as r% Vifortant areas to ; 
assess because over 75% .of- the projects Incited §u^h ^f%i& amons th#ir 
program objectives* The objectives of projects and tljj^ *jatiire of tthe^ 
activities In some projects also made it appropriate - assess whether student 

• ' attitudes in these areas had improved during .the scM* til V<N* * * — esult of 
project exposure* A 

In many projects, a range of culturally related and c* o^n^ellng activities are 
" used to Improve attitudes about school, remaining In " s^fHsnli and students ' 
feelings about themselves, and to measure their knov 1 * ed9e :f Indian-related 
topics. Often, project activities directed toward t&» jiteomes ars quite 
Intensive, though elsewhere the activities areqult% * dffpior Indirect. 



5 s 



Sparti'cularly, Professors Morris Rosenberg of the DPlV%flr«f ffytf Maryland and 
Joseph Trimble of Western Washington University. 

^Examples of such activities. Include :\ providing eJtiVeii fMtrlps-, or" group 
presentations in the area of Indian* culture andhtory pwiding role models , 
through s^efii«T^*eachers, tutors, or ether Indian /staff '| flhipfoviair-ig 
intensive-counseling in academic and vocational arias* . 

JijnioPMBirar associates, ib 
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In some projects, the effects of previous years 1 phlject/eff6vi»^lll^ a )lave,been 
great enough to minimize the chances of successfully measuring project effects 
within the single year of this stucjy. Thus, the stated objectives and the 
nature of the activities In many projects made It quite proper to assess / 
whether student attitudes In these areas had Improved during the school year 
j| $ a result of project exposure. Consequently, the plans "called for students 
to complete a series of attitude scales on relevant demenslons of Interest 1 
(along with other questions deallnfrwlth project participation) during the 
fall and spring site visits. / 



Specifically, data were collected from over 12^000 American Indian and Alaska 
Native students In the fourth through twelfth grades In the fall or spring of 
the 1981 -82 school year, As such, It represents the largest study of Indian 
student attitudes to date, being much more comprehensive than the National 
Study of American Indian Education, which was based on 2,422 students and 
other data sources (Havighurst, 1970). Table 3-3 Indicates the number of 
students who supplied attltudlnal Information In the fall and spring, 



TABLE 3-3 . ' 

NUMBERS OF STUDENTS SUPPLYING ATTITUDINAL INFORMATION BY TYPE AND GRADE RANGE 

■ » 
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Type of Information . 



Student questionnaires 
Student; questionnaires 
Both fall and spring 



fall 
spring 



Number of Students 



by Grade Range 




4-6 


7-12 


Totals 


5.328 


8,149 


13,477 


5,173 


7,366 


12,539 


4,255 


5,665 


9,920 




The development of the attitude scales began with the preparation of a 
literature review on the Instrument selection, measurement, and conceptual 
Issues involved in assessing Indian students* self-concept (Goldsamt and 
HIppSp 1980). All dimensions and items were chosen with care in view of 
cultural and measurement factors which could affect their validity. 



For example, Itenis pencerning the academic sel f -concept domain used in the 
study came from Abdel-Mawgood and Hatch (1 972) and reflected the work of A 
Brookover, Measures of global sel f -esteem came from the widely used work of ^ 
Morris Rosenberg (Rosenberg and Simmons, 1972; Rosenberg, 1965), Other 
attitude-toward-school and self-concept items came from a highly modified 
version of the Ceopersmith Self -Es teem Inventory* adapted from the 
Pennsylvania Department of Education's educational qualfty assessment battery y 
* {1980; Kohr, 1982) and the self -perception scales prepared by Trimble 
(Trimble, 1977; 1983), Core study staff, together with a resource team, also 
prepared items to tap specific topics such as cultural identification and 
pride, .._ 

In addition to identifying Items that measured specific dimensions or topic 
areas, the research team was also concerned with the cultural relevance of the 
Items pmd age-grade levels of the respondents* Thus, after the Items were 
Identified, thty were extensively reviewed /or amblgui^, acquiescence, social 
desirability, response format, appropriateness of age level , and cultural 
relevance. Although the study team recognized the broad cultural differences 
existing among Indians and attempted to design 4§ales which would minimize - 
cultural bias as much as possible, sources of epror imposed by semantics and 
cultural differences may still have been present, v_ 

:[.-.. - ■ ... . ... 

During this development process, f the items were reviewed by over 20 Indian 
educators, representing many tribes In varies parts of, the U.S., and 
including professors, local project directors, and teachers. Also solicited 
and used throughout the development of the Instrument and student data 
collection procedures were the comments and suggestions of the members of the 
study's Technical Advisory Panel, After refining, rewording, and eliminating 
dimensions and items, the scales were pretested on a small sample of 
elementary and secondary school Indian/Native students. Item analyses were 
performed to determine the response alternative and distributional 
characteristics of items, scale internal consistency reliabilities, 
correlations among items and scales, and the factorial structure of dimensions/ 
and items, 

The final dimensions which were retained are presented In Table 3-4. For each 
dimension, composite scores were formed by adding individual item responses 

^ „■„ - ■ ■ ■ - ■ ■ ' " _- MTEI.OPM1NT AMQCIAras, iNi 
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and dividing by the number of valid responses; the same Items were used "in 
both the fall and spring versions, 

. _ . ■ • - A - .y- 

TWo stiyient questionnaires were developed with these dimensions, The grates 
4-6 fornf is a scaled-down and adapted-when-necessary version of the grades 
7-12 form. It measures the same subdimenslons and contains many of tfie same 
Items, to permit cross-sectional trend comparisons of patterns of shift in 



these dimensions across upper e 1 emen ta ry# i$-6 ) , middle (7-9) and high school 
(10-12) grade ranges. The number of Items' each of the nine scales, and 
what dimension each scale measures are presented In Table 3-4. • 



TABLE 3-4 . 

NUMBER OF ITEMS USED TO MEASURE ATTITUD I NAL DIMENSIONS INCLUDED IN THE 
STUDENT QUESTIONNAIRE FALL AND SPRING VERSIONS (GRADES 4-6 AND 7-12) 





.11 . .. rr -— 

/v /■ - 

Dimension Measured 
(no order intended) 

" r i 


Number of Items In 
The Grades 4-6 Form 


Number of 

The' Grades 


Items In 
7-12 Form 


i ' 

1. 


Attitude toward school 


9 




9 


2, 


Value of education 


. 3 




7 


3. 


Academic self -concept 


4 




5 


4, 


Global self-esteem 


6 




a - 


S, 


Sel f*dgrogation 


2 




2 


6, 

7. 


Self-esteem 
School fairness 


■ 4 

> 7 ' 


J 


6 
2 


8. 


Indian/Alaska Native 

Identification with 

ethnic heritage* * 






10 


9. 


Cultural pride/preference 
for instructional settings** 

" : TOTALS: 9 Dimensions 


5 

40 Items 




8 

57 Items 



*A shorter version of this composite index appeared in fall Student 
Questionnaires, * < 



**Not Included In fall Student Questionnaires 
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To confirm thr qua! i ty of the Items and how the deal red 

dimensions, the Internal consistent reli abilities ^nd factorial structure of 
e|ch dimension were separately examined for the fall and spring data bases* 
By '".doing so, four items In each student questionnaire (grades 4-6 and 7-1 2) 
were excluded from further use, since they had inadequate psychometric 
properties, They either did not vary sufficiently or highly correlated with 
stronger indicators of particular dimensions. 

„ . = . . . . ... ... ' ■ ' . ; .•• • . .. 

In order to test further the scales which were developed* correlational and 

factor analyses were also carried out. The resul tr of these analyses , 

togetb&r with basic statistics that describe the characteristics of the 

scales, are presented below, 

(a) Factor Analytic Resul ts . Item factor analyses were performed^for both the 
f al 1 -and spring 4ata sets for both grade ^ 
v versions of the elementary and secondary questionnaires were 1 ncl uded in a 
factor analysis designed to prqduce a factor matrix with maximum 
general liability. Principal components factoring followed by an 
orthogonal Uvarl max) rotation produced the final factor matrix* Only the 
results from the spring data set are reported here, because they are 
comparable to those found for the fall data and they include cultural 
Identification Items not Included in the fall version of the questionnaire. 

The elementary^ grade range questionnaire items produced six distinct 
factors that are described below: 

c Attitude toward school : consisted of variables depicting some 
pbsTtivF orientation "to school, school activities, and teacher caring* 

* Academic self -concept: included orientations of school -related 
performances and se if -appraisal * 

» Value of education: consisted of Items that speak generally to the 
■ importance of education and learning* 

• Sel f -esteem : consisted of personal assessments where respondents 
expressed opinions about themselves in reference to others. 



• Sel f-derogatl on i const sted of variables hiving to do with some 
negative sei f -attribution. 



# Cul tural 'prf.de ; consisted of Items tappin 
culture and pride about one's tribe* 



g some aspect of the Indian 



\ The Items that loaded on the six factors were quite consistent with the 
dlmensi ons 1 dentl fled and establ 1 shed during the devel opment of the 
questionnaire. In this vein* then, the factor analytic results verified 
and confirmed the decisions made concerning what Items and scales to 
retain* and thereby established confidence In the scale reliabilities, 

The spring version of the questionnaire contained a separate set of 
culturally related items. Initially, It was thought that these 1 terns 
wpuld provide Information concern ihg their perceived Importance to 
respondents participating In local programs. However* a more detailed K 
1 rrspectl on 1 ed to the concl us 1 on that they might form separate dimensions 
of their own and might also be correlated with other related items not 
originally Included In the scale. 

These 1 terns , together with other items measuring cultural pride, were 
\ analyzed by factor analytic methods. Three factor patterns were 
produced, Seven formed the dominant first factor and clustered around a 
theme dealing with Identification and ethnic heritage, Five items loaded 
on an instructional dimension, Two other Items formed a relatively weak 
third factor that emphasized a type of ethni c?\ n tegrati on-segrega ti on 
domain. The third was dropped from further consideration due to its 
measurement weakness* but the other two were added to the original six so 
that the elementary age group items tapped a total of eight fundamental 
dimensions. 



A factor analysis of the secondary grade range Items yielded seven factors t 

t Attitude toward school : contained Items that appear to emphasize 
teacher behaviors, school activities, and a positive orientation 
toward school. Hie structure of this factor closely resembles the 
first factor in the elementary group data set, 
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• • Cultural pride s consisted of items that tmphasize pride in tribal 

£ff1 i tati on , l earnl ng about Indians and one 's tribe, and the 7 
Importance with which one's tribe js held, — - 

jfel f -esteems contained 1 terns Indicating ah orientation toward doing ; ( 
things wel l , sel f -satisfaction and posi tlve attitudes toward 
themselves. It resembles the fourth factor of the elementary group 
factor analysis* 

# Value of education: contained Items that emphasize interest in 
classes and learning, doing well, staying in School, and the 

; difference school makis in one's life* The factor resembles the 
pur-item factor wltiythe same theme In the elementary group data set. 

ft Academic self-concept: composed of Items tapping some aspect of one's 
performance in school, ' '. *. 

ft School fairness r appeared to tap a school -rul es and school -climate ^ 
dimension* ~ ^ 

ft Self -derogation: composed of wanting more respect and feelings of 
r usel essness , In content , It resembles the fifth factor in the 
elementary group data iet* 

** • • ■ . . \ ' ■ ■ 

Consistent with the conclusions drawn for the elementary group factor 

dimensions, the secondary group items also appeared to follow the patterns 

identified and developed during the questionnaire development phase of the 

project. The factorial structures of the items lent credibility jo- the 

domain identified for assessment* and, therefore* substantiate the 

internal consistency and homogeneity of L the items used as scales, 

Based on the results yielded by the factor analyses, it was decided to 
form separate scales and use them in subsequent analyses, . The nature of 
each scale for each age group appeared to meet minimal criteria for scale 
construction, The Victor loading for each Item and scale exceeded ,30, 
and the interconnections between scales were low enough to suggest a 
minimum of overlap between constructs measured by these scales, (See 
Tab ley 3 -5 and 3-6 for these Interconnections. ) 

An analysis o^ other cultural identification items was also performed on 
the grade 7-12 group data set, The analysis yielded three distinct 
factors* the third quite weak, and closely resembled those factors 

> - -'- - 

produced from comparable, items In the elementary group. The two factors 
which were retained are? .<;• . 
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r Identi fl cation wl th ethnl c heritage r 1 terns Toad hi ghest on th^s 
factor rer I ecti ng m story or tri oe and culture, 1 earning about 
language and Indians^ going to school with other Indians, and priae in 
tribal affiliatlpn/ / J l 

• Instruction : items loading highest on this" factor deal With school 
activities in which Indians participate, including the kind of 
classroo* and teachers used for instruction. y > 

■ ' v - - \ ; : ■•■ * ■ " ■■ . ' , 

As Indicated above, merging the two types (self and cultural ) of factor 

analyses yielded nine distinct factors appearing in fall and spring grades 

7-12 instruments. The strength of the loadings on each of the factors led 

investigators to form separate scales based exclusively on the distinct 

patterns. 



(b) Composite Scale Results . Items on each scale were then subjected to 
conventional analyses. Tables 3-5 and 3-6 present the intercorrelatlons 
of the attitude scales for the fal 1 elementary and secondary- group data 
sets, respectively- The fall time period was used to obtain "purer" (or 
l£ss potentially affected by program participation) estimates of the 
1 ntercorrel ati on between dimensions or composite scale scores. While the 
correlations between certain scales were moderately positive, most of the 
scales appear to be somewhat distinctive* That Is, the intercorrelatlons 
were sufficiently low at each grade-range. Somewhat different types of 
dimensions seemed to exist arid di^d not warrant, for example, pooling two 
or more scale scores to form some "composite" dimension. 

Table 3-7 presents the Intercorrelatlons of the fall composite with the 
corresponding spring score for those students who were administered bath 
instruments, Standardized item alpha Internal consistency 'reliabilities 
are also contained in Table 3-7 for the same dimensions, so that two types 
of reliability estimates pan be made for each scale, That Is, the* 
fall -spring Intercorrelatlons present an approximation of a stability or 
test-re test reliability coefficient. However, since the fall and spring 
data pere gathered a number of months apart and at opposite ends of the 
school year, the intercorrelatlons are likely to be affected by any 
effects of Pajrt A activities which may have, for example, shifted student 
self -concepts and attitudes toward school. This shift Is desirable, since 
It Indicates program participation effects and Impacts. * 

. ' ' ' 4? • . 
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TABLE 3-5 j , 
INTERCORRELATIONS 8 OF FALL ATTITUDE SCALESi GRADES 4-6 (N»4886) b 





Dimension 


i 
i 


- - z 


% a. 

3 - - 4 . 


c- 


c 7 . '• 


1. 


Attitude toward 
school 


(1.00) 








' ri • . « " 


2. 


Val ue of education 


.38 


(1 -00) 








3. 


Academic self -concept 


.45 


• .24 


(i.oo) 


* 




4. 


Global self-esteem^ 


.29 


.20 


.40 (1.00) 






5. 


Self -derogation 6 ' 


.09 


.05 


.18 .59 


(i.oo) 




6." 


Self-esteem 0 


.31 


.22 


,39 " .91 


.20 


(1.00) 


7, 


Cultural pr1de d 


.13 


.06 


•09 - ,n 


.01 


.13 (1.00) ' 



■ ■ N % . r 

^Correlations greater than .02 are statistically significant at p<. 01 .with 
an N of 5000. ■ » ' . • J 

b Jntereorrelatiens are based on students who had .scores on all dimensions, so 
that relationships could be more readily cornered. 

Correlations between global self-esteem, and self-derogation and self-esteem 
scales are high, because self -derogation and self-esteem are subseales of 
the global scale and contain the same items. The direction of negatively 
worded items, such as self -derogation items, has been reversed for scaling 
purposes; hence the positive correlation of polar concepts, 

Represents the Initially conceptualized dimension and corresponding Items, 

Note: The Value' of (1.00) along the diagonal represents a correlation of a 
scale score with Itself . 
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.;, TABLE 3-6 7 . 

IMTERCORRELATIONS a OF FALL ATTITUDE SCALES % GRADES 7-12 (N"7547 f ) 





Dimension 


1 


E 


4 


4 . 


e 


0 


'J 


9 

Q . 


1. 


Attitude toward 
school 


(1.00) 
















2. 

Sr. 


Value of education 
Academic self -concept 


.58 

.28 


(1.00) 
.38 


(1.00) 












4. 


School fairness 


..29 


.33 


.17 


(1.00) 










5. 


global self-esteem 0 


.24 


.29 


.47 


.19 


(1.00) 








6. 


Self -derogation 0 


-.07 


^ -.03 


, -04 


. .1o 


,49 


(1.00) 






7, 


Self -esteem 0 


.31 


.35 


.52 


.17 


.90 


.08 


(1.00) 




8. 


Cultural pride c 


.08 


.15 


'.07 


-.07 


.07 


-.16 


.16 


(1-00) 



( ^ 

^Correlations greater thin *03 are significant at p f < .01 with an N of 7000. 
Intercorrelations art based on students who had scores on all dimensions * so that 
relationships could be more readily compared* 



^Correlations between global self-esteen^and self -derogation and self-esteem scales 
* are high, because self-derogation and self-esteem are subscales of the global seal 
and contain the same items. The direction of negatively worded Items, such as 
self-derogation items; has been reversed for scaling purposes; hence the positive 
correlation of polar concepts. f 

^Represents the initially conceptualized dimension and corresponding items. \ 

Note: The value of (1 .00) -along the diagonal represents a correlation of a scale 
score with itself* 
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school 
Value of education 
Academic self-concept 
School fairness 
Global self-esteem 
Sel 

Self-estiem 

identification with b 
ethnic heritage . 

~ " ^ " b 



es 



5,039 
5,235 
5.172 



\ 5,2)8 
5,199 
5,161 

' 5,159 



6RAPES b-1 : 

Standard Estimated < 
Alpha Tist-Ritfist Fall-Spri 
tMfc^ > Mtt; f '(CT) 

.n - .63 



.46 V 

,62 
* 

.51 
.51 

.47 

,75 



.37 
.53 

.42 

,M 
,38 

.66 



.28 

.47. 
t 



,29 



I 



5,140 .56 



.47 



.79 
.76 
.73 
,64 



7;853 
8,050 
8,034 
7,988 
8,005 ,68 
7,937 .39 
8,018 .76 

7,3)6 ,83 



Standard Estimated 
Alpha :" TM&i|i||ist' " . ■ 



.70 
,67 
,64 
.55 
.59 
.30 
,67 

,74 



7.271 ,85 



.76 



.59 
.60 

.67; 

,42 
,54 
,37 
,53 



-Intercorrelations are based on students having scores on all dimensions, so that relationships 
compared, All intercorrelations in this table are statistically significant at p< .01. 

^ Raprasents the cul tural dimensions developed from factor analyses and corresponding Items. * \ 

r". ' - '. ■ ■ ' ■ ■ . ' " .. • ■ ' . • . [ ' 

c Standardiiid item alpha internal consistency reliabilities are reported for fall data for school and self-related 

scales, and for spring data for, cultural identification/pride scales, which IThelr two-factor form, ware not ad- 

ministirid in thrfall, 

dn.»j ,„k**>*H»n i r^arUnn fartnmf no htm tb alnha reliability, as reeoiiended by Murray (1983), 



•'*.' ". ;■ ' " j v =*\ ■ - ." ^V-:' : 

In general, considering: (I) the earlier findings that Items intended to - 

tap a particular dimension actually did so (judging from the factor 

analysis results) and (2) the underlying issues which affect measurement 

of Indian student's perceptions of themselves and their school 

experiences, the level of internal consistency and stability found across 

these attitudlnal scales is fairly respectable, However,* they are lower 

than fully desirable (I.e., if they had alpha levels of ,80 or above). and, '< 

therefore, the resul ts must be treated id th caution* /Elementary grade 

level rel labilities, particularly, tend to fair In this latter category. 

The reliabilities at the grade 7^12 level are higher than for the grade 
4-6 instrument, also reflecting the greater number of items used to 
measure virtually every dimension. However, given the fact that other 
studies have obtained higher reliabilities than the present one, findings 
based on som<# of the scales should be 1 nterpreted with caution, as was 
done here. Hence, greater emphasis has been placed on th^ relative levels 
of attitudlnal scores, thin on whether or not they tended to reliably 
shift over the time span of aP single school y#ar, 

, ■ . ... . . = ^ ..... 

In addition, attitudlnal results have been presented at several levels, ' * 

.The percentages of Indian studerfts who- completed the items comprising each 
dimension are given for the elementary and secondary grade ranges for the 
fall ands spring time periods/ These analyses are presented for each time 
period separately (I.e., as cross-sectional comparisons) so that overall 
student attitudes at two district points in the school year can be 
examined. This Information was also important at a descriptive level in — * 
order to assess the context in which projects operate, Interpret other 
student-level data, and assess the appropriateness of citing a change in 
these dimensions among the projects' major objectives. Group mean 
attitude scale scores were also presented, both overall and in terms of 
meaningful differences found when using particular program variables as 
cross-break factors, Finally, the relationship of attitudes to academic 
performance was presented. 
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Addi ti onal Conslderatl ons 

Finally, several cornerstones of the overall design are relevant to mention. 
All instruments represent the result of extensive review* -for content validity 
and sound measurement properties , by a Targe number of Indian educators and 
recognized eval uation authorities.. . (See Appendix 2 for their names and 
professional affiliations-) In addition, the sampling design was constructed 
to provide statistically representative samples of projects, parents and 
students, and tribal leaders or^community representatives. Also, relatively 
more of the larger Part A projects were chosen to Increase the likelihood of 
detecting impact, since more resources were presumably available in such 
settings. \ > 

Furthermore, the impact analytic approaches deliberately incorporated various 
contextual , project, and student characteristics, so that such variables as 
project size and location were directly considered as alternative or competing 
explanation^ for Impact levels and shifts in those Impacts. The analyses also 
proceeded in a series of stages or modules, which went from purely descriptive 
to intermediate' and advanced impact analyses* and to multivariate analyses as 
warranted. This modular approach extended to jointly analyzing impact^ 
indicators, that addressed dimensions related to each other. In this way* 
obtained findings and Inferences were made more clear-cut. The Interpretation 
of the findings was enhanced by using external sources of data for baseline 
purposes, In addition, using relevant data from past or ongoing national and 
local evaluations and studies helped to place the study findings In context 
and substantiate their validity. 

in summary, the special design Issues which emerged in evaluating the Part A 
Entitlement Program were problematic. However, they have been resolved by 
using a variety and breadth of design approaches and appropriate statistical 
and interpretative techniques. Such a combination has produced a descriptive 
yet in depth profile of what Part A projects have accomplished thus far and 
what remains to be done to heTp Indian students. 



Sampling Plan " .. 

1 9 Study Popul a ti on and Sampl e Si zes 

:'.:V . V. . : *■ 0.." - -V.'. '.:V ■ ' : '. 

Because the Part A evaluation study has several purposes, a fairly complex 
sampling design was jjsed to select projects, school districts, and 
respondents , The design was Intended to f 1 1 'the study scope (shown 1 n Table 
2-1 In Chapter 2), particularly the Information-gathering needs of the Local 
Projects Impact Study and the Alternative Resource analysis facets of the 
\ overall study design. . S 

t . ■ ' » . • 

• ■ . ' ' ' ' \ - -' i ' 

The study Included all public school districts that Were eligible to receive 
Part A Indian Education Act (P.U 92-3T8) 7 funds. In 1980 s? the Department 
_ of Education estimated 3,177 school districts, other than tribal ly-control led 
school 17 were elYgibl e to receive such funfis* (f rlbal ly^controll id * 
schools, 8 which are eligible to receive Part A funds, were outside the scope 

of this study. ) r - 

-.. . ■ ■ -. *■ * ■ • 

Two populations were surveyed: 

' - r ■ * * ■ ■ * * - 

• For the sample of school districts from which data on Alternative 
Resources were collected, the survey population included all 3,177 school 

4 districts eligible for Part A funding. 

• The survey population for the local project impact evaluation was a 
865-project subset of, the 3,177 school districts Call of which were funded 

- for at. least 3 years (1979-81 ) and included more than 30 Indian/Native 
students in 19803* "'■ ■ 



4. 



^Eligible districts are those in which ten or more American Indian or Alaska 
Native students are enrolled, except In the states of Alaska, California, and 
Oklahoma, in which all districts enrolling one or more Indians/Natives are 

.eligible, i 

8 There are only 45 tribal ly-control led schools participating In the program, In 
many respect* they are quite different from public LEAs, ahd it would t be 
inappropriate to study them in the same fashion* They may be the subject of a 
separate study at another time. ^ 

. ' - r. • . 54 •• ' 
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In nearly all eases, there was a one-to-one correspondence between a "project", 
and a "school [district." In a few cases, a project was comprised of two 
school districts that jointly operated the project. The Impact palliation 
study pertained to a survey population of 865 projects. The 865 projects were 
a subset of the 1 ,052 projects funded In 1981 that were also funded for at 
least three consecutive years (1979* 1980, and 1981) and which reported more 
-than 30 Indian students In 1980. A sample of 119 projects was originally 
selected from the 865 projects for the Impact study. (See the next section, 
"Sampling Design.") f ; : \^ 

Of this original sample of 119 projects, 19; declined to participate in the 
study at the outset. With one exception fa project in Maryland), each of 
these projects was replaced by another randomly selected project from the same 
sampling stratum. Reasons for non-participation Included: recent death of a 
project dl rector, recent turnover of key project staff, a cri si s of some sort 
In the district unrelated to Part A but precluding cooperation of district 
staff, and district policies discouraging participation in federal studies of 
any kind. Thus, 118 projects were visited In the fall of 1981 and, of these, 
114 cooperated fully with the study throughout. Two projects withdrew from 
the study between the fall and spring visits. The other totfd^pr© jects refused 
access to a random sample of their students and various data about them, 
Three of these projects (Bering Straits, Alaska; Denver, Colorado; and San 
Diego, California) were not revisited in the spring and were excluded from all 
analyses. Some data from the other project (El Reno, Oklahoma) were gathered 
in the spring, and this project was included In analyses wherever possible. 
The final sample size, therefore, was ,115. Analysis of the characteristics of 
the non-participating projects indicated that no bias threatening the 
generality ©f the findings Is likely to have occurred, either from the Initial 
refusals to participate or subsequent attrition. . , 

By including only projects with three or more years of Part A program 
operations, "start-iip" Implementation problems, which eofifound and reduce the 
likelihood of program effects, were avoided. Also, longitudinal data related 
, to student effectiveness and other factors were more apt to be present- 
Similarly, Including only projects with more than 30 students assured a data* 



base that was sufficient for assessing student effects and a local project 
adtalnl strati ve structure sufficient to respond to stu^ Information needs* In 
1980, projects with more than 30 students end three, or more years of program 
operations experience represented over 91$ of all funded LEAs, Thus, only a 
small proportion of LEAs was screened out, and the^roportl on of Indian 
students in projects screened out was also not of major concern. 

The sample for the Alternative Resources Study represented a combination of 
two samples: (1) the Impact evaluation sample of 119 projects representing 
the subset of 865 projects, and (2) a sample of 108 LEAs selected to represent 
the population of 3,177 LEAs that were eligible to receive Part A funds, but 
were not Included in the subset of 865 projects. These two samples, properly 
combined and weighted, constituted a single sample representing the 3,177 
eligible LEAs, This sample was used to prepare a resource analysis based on 
estimates of three types of information: * 

• Total federal education funds expended by local school districts on Indian 
students; - 

• The number of Indian students served by these funds* and 

• The types of special services the students receive, 

2, Sampling Design for Selection of Projects for the Local Projects Impact 
Evaluation . 

For the Local Projects Impact Study, 119 projects were selected, but due to 
attrition, sufficient data for analyses were obtained from 
projects. Within the sample projects, data were collected fro« an appropriate 
district administrator, the project staff, members of parent conml ttees , 
parents of students, student records on file, and students,' Multistage 
sampling, particularly the sampling of students from whom to obtain, 
information, was Involved for much of the evaluation data, However, the 
primary sampling units were projects, v 

Very briefly, the sample of local projects may be described as a stratified 
random sample with pdRbilltles of selection that were proportional to an 
appropriate measured project size. One project was selected from each of 

119 strata, yielding a sample of 119 projects, 

^ — sssssai" . ■ development associates, INC, 
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One of the first sample design decisions to be made involved a choice of 
prfthaMHtle^ for selecting ^ 

data to be collected called for selecting projects with equal probabilities 
regardless of size of project. For other data, especially where Individual 
students were the units of analysis. It was preferable to select projects with 
probabilities proportional to the number of Indian students. y- 

General experience with the statistical efficiency of alternative measures of 
size as a basis for sample selection* and consideration of various kinds of 
data to be collected, led to a decision to use the square root of the number 
of Indian students as the measure of size, and to select projects with 
probabilities proportional to that measure. Thus, a project with 400 students 
would have a probability of selection which was twice that of a project with 
100 students. 

To offset the higher probabilities of selecting large projects, smaller 
sampling fractions were used for sampling students within large projects. In 
fact, the procedure was to determine and apply w1 thin-project sampling 
fractions so that all students In the population of 865 projects would have 
the same chance of being In the sample, regardless of project size. 

Stratification of projects. Thirteen very large Part A projects with more 
than 2,000 Indian students were treated as a special group. Three of the 13 
were Targe enough to justify being included in the sample with a probability 
equal to one. The remaining ten were stratified into four strata, and one 
project was selected with a probability proportional to size (i.e., the square 
root of the number of Indian students) from each of the four strata. This 
left 852 (i.e., 865 minus 13) LEAs to be sampled. These 852 LEAs were divided 
into 106 strata, approximately equal In "size," and one LEA with probability 
proportional to size was selected from each. 

One of the design objectives of stratified random sampling is to achieve a 
high degree of homogeneity within strata. To accomplish the stratification, 
12 geocultural regions and three density groups were defined, as follows: 

57 ' ' . 



9 

ERIC 



45 



• Density : Projects wert first classified by Indian student density, that 
is, by the ratio of Indian students served by a project to the total 

^jUBQjir^ ojLs tudentt^ 

degree, this variable stratified the population 1n terras of: • 

Urban projects that have diverse tribes and relatively few 7 
Indians/Natives compared with the total number of students In the 
district; and v ' . , 

Reservation-based projects that have a single tribe and a high number 
of Indian students compared to all other students. 

This variable was considered Important since the nature and level of : 
student density could affect the extent of project Impact- For example, 
Indian/Native students In urban situations may have few opportunities to 
strengthen their knowledge: about their Indian {tribal ) heritage, 
culture, and history. This Is a particularly important factor 1n areas 
with large numbers of relocated Indians/Natives who are often <* 
disconnected from traditional family and tribal supports, especially for 
their children. Also, urban districts, unlike most reservat1on-*ased 
projects, generally serve more than one tribe, and thus serve students 
who are cultural ly diverse. Three density groups based on percentage of 
Indian students were established, based on consultations with Indian 
educators: less than 5$, 5/01*70%, and 70.01-100$. 
* 

• Seocultural region: This was the other kty stratification variable, 
Projects were stratified by 12 geographic regions as shown in Table 
3-8, Projects were listed by density group within each of the 12 
geocul tural regions. This list included 852 LEAs, as the 13 largest 
were treated separately as indicated previously. From the listing of - 
projects by density group within regions, the process of stratification 
continued with the objectives of equalizing strata by size and selecting 
one project from each stratum. 

To form the 106 strata, In some cases, project 1/*f a particular 
population regional density group were broken town further, In other 
cases, some combining of projects (e.g., across regions) was necessary 
to obtain a sufficient number of projects to form a stratum. For ~ 
example, suppose the number of projects In the high density group in a 
region was enough to form 1 3/4 strata, One stratum would be formed 
within that region and density group, and the second might have been 
defined by Including some nearby projects in another region, or perhaps 
some projects of highest density within the middle density group in the 
same region. 1 * - 

In a few cases, strata were quite small In order to avoid wide cultural 
differences within a stratum, and to assure that certain tribal groups 
would be represented when a random selection was made. The regional 
breakdown not only divided projects* Into geographically distinct areas, 
but also into cultural regions as well . Use of geocul tural strata 
ensured that diverse tribal affiliations and parts of the United States, 
including Alaska, were Included in /the sample, 



58 ' . 

• • ■ ., ■ ■ DEVELOPMENT ASSOCIATES. INC. 

ERIC 



TABLE 3-8 



GEOCUL - TURAL REGION CATEGORIES 



The five Office of Indian Educat-f on Technical Assistance Center Regions were 
subdivided as follows? 



1 awafle, New Jersey, District of Cel umbla » Tennessee , North Carol 1 na, 



./ South Carolina, fiiorgia* Alabama, Mississippi* and Florida, 

3. Eastern Region 2 Wisconsin, Iowa, and Minnesota. 

4. Dak otas (Region 2)-"Noorth Dakota arid South Dakota, 

5. Western Regiorf 2 - Nebraska, Wyoming, and Montana, - 

6. Alaska (Region 3) 

7. Southern Region 3« WasBtington, Oregon, and Idaho 

8. California (RegtdH) 

9. Southwest Region ;4- Arizona and New Mexico 

10. Northern jtegion 4*" Nevada, Utah, and Colorado 

11. Oklahoma (Region 5) 

* 

12. Region 5 Remainder ■» Tescas, Louisiana, Arkansas, Missouri, and Kansas. 



1. ) Northern Regional > IlUnois, Michigan, Indiana, Ohio, Pennsylvania, New 
York , Connecticut; Rhode Is! and, Massachusetts , Vermont, New Hampshire, 
and Maine. 




After making a pr — elfminary selection of the projects, the composition of the 
sasnple Wdiriylgwved end discussed with the study 1 s Technical Advisory Panel of 
Indian educators on the basis of the foil owing factor — s : 



• Geociiltural r— eglon (overall for specific states* and within particular 
statesiuch i i Alaska, California, and Oklahoma); 

, / . = - ■ • - ' ' 

• Proportions o reservations, rancher las, urban se -ttlngs, and suburban 
sites; ■ 

i w 

• Indian tribal diversity and heterogeneity; and , 

• Shifts from li^JSOto 1981 in the number of student s served by each project. 

J • 
Based on these fa* <:tors, some strata were redefined an- d some resampling 
occurred* However ir, the number of strata and the samp~Ie size were not 
changed*" The ffn^al selection of projects adhered to "the principles of 
probability sampl — fng, except that substitutions were later required owing to 
non^eooperitfon o— f some LEAs, When necessary* a subs — tltute was sel ectcd which 
matched UHrlglwial as closely as possible In slit aand other 
characteristics* The composition of the sites which tntfere chosen is presented 
In Tablet 31 

Mathematically* tSfie probability of any given* project's being selected was s/S, 
where ,| s h lithe square root of the number of Indian students In the project , 
and "S" 1 s the %u^m of the square roots for all projee#=fcs In the stratum from 
which It vyas sel aerated, When a substitute was selected^ "s" became the size of 
the substitute* nrat the original selection* 

3, Sampl IngLg^esp jor^i dents Within the Local Projects Impact Eva^ff^tlo n.S jtiiple 

In general, th# following ten respondent groups were fffnvolved In the data 
collection which (Occurred in the 115 randomly selectee^ impact evaluation 
projects*. - 

• School ^itrl^t aWnlstrators (superintendents or— their designees* and 
assistant superintendents, coordinators of federal programs, or another 
administrator directly supervising the Part A project director); 

• - 60 
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TABLE 3-9 

SELECTED CHARACTERISTICS OF S/MPLEB IMPACT EVALUATION SITES 



I. Location (N*115) N % 

1. On or Near a Reservation 44 .38.2 

2. Other Rural 31 27,0 

3. In an Urban setting (10,000-50,000)" 16 13.9 

4. Metropolitan (50.000 *) ' 24 20.9 

II, Geocul tural Region (N»115) ...... 

Northern Region 1 13 11.3 

Southern Region 1 6 5.2 

Eastern (Region 2) 8 7,0 

Dakotas (Region 2) 6 5.2 

Western (Region 2) 7 6.1 

Alaska (Region 3) 7 6.1 

Southern Region 3 * ''.*.. 11 9.6 

California (Region 4) 12 10.4 

Southwestern (Region 4) 14 12.2 

Northern (Region 4) 3 2.6 

Oklahoma (Region 5) 25 21,7 

Remainder (Region 5) , ' -3 2.6 

III, Percentage of Indian Students In the LEA (M^S) 



.0001 - 5% 40 • 34,9 

5,0001 -208 25 21.7 

20,0001 - 70.000S 28 24.3 

70, 0001 % & over > 22 19.1 

IV. Total Number of Indian Students Enrolled (N=11 5) 

31 - 99 20 17.4 

100*- 219 ■ . 35 30.4 

220 - 549 32 27.8 

550 - 999 ■ U ' . 12.2 

1000 i, over w 14^ 12.2 

V. Project Components (N-115)* 

1. Have a Tutorial/Academic 91 79.1 
Component (Includes basic skills, 

gifted and talented, etc.) 

2. Have a Counseling Component 60 52.2 
(Includes career/vocational, 

substance abuse, psychological, 

etc.) ' 61 
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TABLE 3-9 (Continued) 

• .; ; . ' ■ 1 . . n . .' ' . :m j^ 

3- Havi a Cu1 tUM*"«1 Component 75 65 ,l 
(Includes language, cul ture, 
history, curt-em nt affairs, arts 
and crafts., *t*sC.J_x* 

4» Have a Formal ■ Home-School 51 44 J 

Coordination Co-omponent , ' 

5_ Have a Parenta".! /Student 28 24 J 
Costs Compon^n- t 

VII, Parent Cownlttee (i N=115) 

1. Number of memb»~ers 

Total: 3-9 58 * 50 J 

10-19 46 40"J 

20-29 5 4-3 

30-65 .6 5 J 

1-9 69 60 J. 

10-19 38 33 J 

20-29 2 U 

30-65 6 fij 

Non-Indian; 0 58 50-4 

1-9 57 49 J 



\ 



♦Per-eenfcages may total moere than 1 001 due to multiple coinpqponents within projects, 

... ' \ 
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•^Ptrt A project directors; 

■ 4 Teaching and non -teaching project staff (fcliils - tteh IhcWfng eoutJiselers, 
resource specialists, and tutors); 

• Parent comnittee chairpersons and three active ^coiisiift^inemb^r^ 
designated by the project director or chat ijirs^oii j 

• Indian community and tribal leaders (one jyffson note dfrietly wso^clated 
with the project selected 1n the foil owing ofdewr, depending on\ 
availability; (a). chairperson of the tribal tduocat-f ofl «iitt%e ohm* his/her 
designee, (b) chairperson of an urban Indl iH£m%tmf kl £ -education' CQ©mmittee 
or his/her designee, (c) person identified inmost interviewed panrents as 
the most Interested and Influential rnenibe^ if t»ie ludf iNoninuriltJ^ with 
respect to educational Issues); 



• Indian students In grades 4-12 attending ^cW=s and fn fades Whsere Part 
A project activities were available; 

• Parents of a random sample of sampled Jridi eistcu den t^i 

• Principals of schools having Part A actlytliis; 

• Regular classroom teachers of Indian studyib t>OJt n^t associated w**1th Part 
A; ahd 



• Students formerly in high schools in sanp-M-df stH^t^ '(I.e., those having 
Part A projects). ) 



0 
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Each respondent group therefore became a categ^isfT pevfbMl or indfWvl duals 
represented by: the school district^ Part A pgf&jertt, iMt aficomijunltyw, 
students participating in Part A activities* parents of* such students. 
While most groups of respondents were contacted ihrfl n{| W spring of tehe 
school year, a few were contacted during th§ f#lli» well* 

The procedures used for selecting each of thes^typras oT Mpondents^ along 
with a description of the type of information tftafnaed, #NsymmarlzecM below, 

% 1 I « *■ * 

(a) School District Administrators. In eich otftto 115 r**ihly selfc^ted 
projects, the corresponding school d1$%Nc&kpmmri n\gt\Mt or the 
coordinator of federal programs (or adfojini^tratwr who directly sup*»erv1sed 
the Past A Program within the district) wa#eoi|fc0Ct^ 

While the title of the supervising admin Is^Mtpr— vaMMfrom distr— let to 
district, In reality, the role of the indlyW anq h-fi/her acces.is to 

V <■ : ■ V « . 



project data were the, key factors for selection. Probability samp ling was 
■ •■ not involved in selecting the type of actainistrator. Rather, the 
administrator considered most appropriate for dealing with the topics 
Included in the LEA Adnlnlstrator Questionnaire was contacted. 

Version 1 of this instrument dealt with background characteristics of the 
: district^ the district support provided to the project, and an assessment, 
of project impacts on students. Version 2 dealt with issues relating to 
the varioclS effects of the project during the school year as perceived by 
the district, as well as the functioning of special programs, such as Part 
A, within the district environment. Version 1 was administered to the 
superintendent 9 the assistant superintendent, or the federal program 
officer during the fall data collection period, f While Version 2 was used 
during the spring data collection period* every attempt was made to 
Interview the same individual as in the fall., 

(b) Project Directors , The directors of the 11 S projects selected or their 
designated representatives, such as a full-time assistant project director 
when the nominal director was only minimally Involved, were contacted 
during both the fall and spring data collection periods. The fall 
Interview focused on project characteristics, while the spring interview 
focused on the project's Impact during the current school year and since 
Its initial funding, 4 Various types of impact dimensions were Included in 
the Project Director spring form, In addition, those portions of the 
Overview of Project Scope and Component Description form that were, 
appropriate to a given project and that characterized the project's 
activities were completed by the director ©redesignated staff during the 
fall visit. Also during the ^all , the field staff completed document 
review and file search activities, using the Document Review form and the 
Post High School Follow-up Survey Questionnaire, The latter was designed 
to record factual Information about the academical ly-rel a ted act1 vi ties of 
a sample of high school sophomores in the years 1970-71 , 1972-73, 1974*75, 
1976-77, and 78-79. These data were gathered from school records, or from 
knowledgeable school staff, family, or friends as appropriate, 
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(c) Project Staff, Projeftct staff were chosen on th© basis of their relevancr 
to th© topics included In the Project Staff Questionnaire and the .... 
partf ciilif project components f or whi ch they Were responsible. In 
general, ill paid stmff who worked at least half time Were surveyed* Kng ^ 
staff mtmber In charg»e of a program component was also contacted, V 

Exceptions; y'j;i 

; i : ' . • • 

# Snail projects : In small projects with only part-time staff $ and wftH V 
no "one, exciudingp the project director* working half time or more* tha 
onifor two non-clf arical staff members who worked at least 20%' of ful l 

(supported f^n whole or part by Part A) were asked to fill out 
staff questlonnai? res. ■ * 

• Larie projects ! _ In projects with ten or more non-clerical staff 
Ming half t1m^ or.more, and at least partially supported by Part A, 
jTwl e of ^staf members was sel ected by the project dl rector based — 
upon a l ist of ttae numbers and types of staff provided by the field 

staff • • * - . 

Project staff were glfven the Project Staff .Questionnaire during the spring 
data collection period. Some were also asked by their project director 
during the fall to complete certain sections of the Overview of Project 
Scope and Component Description form, if relevant to their project 
assignment.- -* 

(d) Parent Comlttae Chairpersons and Members , Chairpersons and two other 
officers of each ©f the 115 Part A parent conmittees were contacted, 
Chairpersons were 1 n "fcervi ewed i n the fal 1 . The two officers and 
chairperson were surweyed, using the Parent Committee Questionnaire, 
during the spring datea collection period. If no chairperson was 
available, vi ce-chai i — persons were contacted. If no designated chairperson 
or vicHhairperson currently existed, the most senior or experienced 
mtrnbar of the commi tteee was contacted, 

% ■ 

* ■ ' * i 
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(e) American IndlaiWAIaska Natl ve G^ommunl ty Representatives and tribal 
Leaders. Thess respondents wotre Interviewed during the spring data 
collection perlcd using^the Inedian Tribal and Community Leader Interview 
Guide. As with parents, this cdata source was considered highly useful for 
determining: '(a) whether thos^f outside the project administrative 

- structure felt that the studen«:s being served benefited from project 
activities and (b) whether the project accomplished certain culturally 
related goals considered impor^Btant to the Indian comnunlty and leaders. 

One communi ty representative or— tribal leader was to be Interviewed in 
each of the 115 local project settings. Tht Identity of the most 
appropriate individual to inter— vfew was not always readily apparent, 
especially in urban projects wf~iere Indian community leaders may not be 
easily visible* and in multftrS bal settings where a number of leaders were 
suitable to interview. Tfierefc^re, it was the responsibility of the data 
collector to determine pnslte Mhom to interview, based upon the following 
criteria: 

• The chairperson of the trltoal education committee or his/her designee; 

• The chairperson of the yrban Indian cfriter^Keducatlon committee or 
his/her designee; or " . ■* 

' / # If neither of these was pr^=sent, the person Identified by most parents 

as the most influential Incdi an individual In the district concerned 

with education! who was notz; employed* by the project or otherwise 
Included in the survey, 

V . . . . 

If two or more distinct tribes or factions were present within a school 
district, a representative froira each was selected, 

(f) Students. j Probability samples of students were drawn in 'each of three 
grade level ranges: 4-6, 7-9, and 10-12. (The grade range of Pre K-3 was 
not considered suitable for stL«c(y purposes,) That is, for sampling 
purposes, each grade-level ran^e was treated as a separate population to 
be sampled, The objective was to obtain a sample of students within each 
grade rangi, so that every eligible student within the grade range in the 
entire population had an equal chance of being in the sample, , For various 
practical reasons that could mmot be achieved fully, 
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t Probability forgMMl a — sampling students within projects* The 
following probafc^D^Uy formula was applicable. for sampling students 

within projects, It was applied to each grade range separately. The 

formula is: ^ • 

where F<i 1s th^ overa^ probability which the jth student in the 1th 
project had of fct>e1ng In the sample, P^ Is the probability which the 
1th projeet had of being selected, and f y is the probability which 
the jth student In the 1th project had of being selected, given that 
the project had already been selected. 

Within a grade rwange and project, all students were to be selected with 
equal probabll 1%-ties. Therefore, f,-,- may be simply expressed as fj, 
the sampling fraction to be-appl led* 1 within the project In question. - 
Also, In this csase^j became F,*. r 

Since Fi was cotwistant for all students In ja grade range, then all 
students in the population had the same chance of being in the sample, 
Si nee the sampl 1-f ng was such that Ff was constant, that constant^^^^ » : . ^ 
became, 1n effecact, an overall sampling fraction that Was applicable to 
the entire population. Three constant values of F were .determined, one 
for each of the three grade ranges. These values of F were determined 
with arHexpeeta"*-t1on ..that a- sample of at least 4,000 students In each 
grade range wouTld be obtained. Then, for each grade-level range,- 
values of fi (w— 1th1n project sampling fractions) were computed as 
follows: ' . 



F 

fi - _ 
PI 

Where , 

F is the overall sampling fraction; 

Pj, as explained above, is the probability which the 1th 
project 4 had of being in the sample; and 

fi is th»e sampling fraction applied to the 1th project in the 
sample. 



• Sampling within schools . In general, each project in the sample 
presented a different sampling problem, owing to varying numbers and 
sizes of school: s and grades at each school. In projects where all 
schools were Inocluded in the sample, the values of fj, as calculated 
above, were dl r* ectly applicable to students within all schools in the 
project. In pr**-ojeets where a sample of schools was selected, the 



wlthln-projfct sampling frictions had to be resolved into within-sehool * 
,* sampling fractions using a probabil Ity formula similar to the one 

above. When a sample of schools was selected, the schools were 

stratified, and one school was selected iProm e 

a stratum contained one school; e.g., all students fn a grade might 
/have been attending the same School , ,. Such a school had a probability 
/equal to one of being in the sample. (It is not feasible to discuss 
/ the numerous patterns that were found and how the sampling of schools 

was performed In all kinds of situations*) ^ 

In general, for any given grade, there were' at least two schools in the 
sample, unless all, students 1n the grade went to the same school . 

It was not possible to assemble In a central location lists of all ' 
eligible students In sample school s* Thus, sampling within schools 
needed to be very simple. The sampling fractions In decimal form were 
translated Into Integer fractions j for example, .15 and .39 became 1/7 
and 2/5. In general, the cal cu1 ated sampl 1 ng fractions to be applied 
within schools were larger than 1/10, and many were 1/2 or larger. 
Considerations Involving simplicity and public relations often led to a 
decision, especially when the sampl ing fractions' were 1/2 or greater, 
to Incl ude al l "el igible students In i sampl e school V That ir, In sPme 
schools the sampling fraction within the school was equal to one rather 
^han to the fraction calculated. 

The sample of students was chosen from the roster of students being 
served by the project. In most Instances, this corresponded to the 
total number of Indian/Alaska Native students certified as eligible for 
service through having Indian Certification Forms (01 Form 506) on file 
with the Part A project director's office. In some cases, the number , 
of students being served differed from the total number of 506 forms on 
file. In such Instances, the sampling frame of students being served 
by the project In each of they'threeNflrade level ranges (which Includes 
the Form 506.1 istings) was the baslsjfor sampling. 

« Al 1 owances for a ttrl ti on . The actual sample of students was drawn at 
the onset of the school year -- inmost eases from a master roster that 
was supplied by the local project jnd classified by school and grade 
level. The sizes of sample chosen* allowed for attrition expected, 
during' the school year. In some projects, the attrition rate was . 
expected to be larger than others. Therefore, project directors were 
contacted before the fall visits and asked to supply estimates by grade . 
level range of the magnitude of attrition rate In that setting. These 
estimates were used in the local site sampling of students to help 
ensure that desired sample sizes would be achieved. 

• Data collected . The types of information gathered from students varied 
with the grade-level range and time of year. All sampled students 1n 
grades 4-12 received Parts I and II of the Student questionnaire during 
both the fall and spring data collection periods. These sections 
gathered information about student participation In project activities 
and ratings of school climate, attitude toward school, Indian ' 
identification and pride, self-concept, and rel ated dimensions (all 
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known to be focuses of Part A projects). These sections were used on 
.'• two occasions so that pre- and post-school year shifts in these 

dimensions could be measured and correlated with the students 1 reported 
participation in Part A Program- activities. Thus; those students 
reporting little contact with program activities became a natural 
comparison group with those reporting relatively moderate or more 
extensive contact with Part A. Data from students who had participated 
In certain types of activities could also be compared directly with * - 
data from students who had been 1n other activities. 

In the spring, those students considered to have participated 
sufficiently in Pact A activities, based on Part I item responses , 
i\ received Part III, 9 which dealt with effects related to culturally 
related topics and activities. . Students in grades 10-12 involved with 
project-related counseling activities (career or academic) also 
received Part IV, Which was a brief set of questions pertaining to 
'their knowledge of post-secondary opportunities. 

m Sample si zes^- With respect to the sampl e sizes being used, 
approximately 5,000 students per grade-1 evel range were Included In the 
fall data collection* Owing to attrition of various types, the number 
contacted in the spring period was reduced to approximately 4,000. 
.. Thus the sample for the latter period corresponds to about 33 students 
per sampled project per grade-level range, or about 11 students in each 
of the targeted grades* 

As indicated, the initial sampling fractions or sizes reflected student 
-dropout and attrition. It was expected that approximately 4,000 
students inr each grade range would be given questionnaires during the 
* spring data collection period* (Attrition by type of students was 
considered In the Impact analyses so that results were not 
arti factually attributed to project Impact*) The combined effects of 
attrition (particularly at the high school or grade 10-12 ranse) and 
subsampl ing of students not receiving certain project components * < 
reduced the total number of sampled students receiving the sprfng data 
— collection forms by approximately 20%. About 4,000 students per grade 
range comprised the total data base of students having both pre- and 
post-school year measures on Part I of the Student Questionnaires* 

Overall, If a project was serving each of the grades 4-12, the total 
sample size would be about 100 students per sampled project* None of 
F ^ these sample sizes seemed excessive. Instead, they represented fairly 
small numbers of students from any grade-level range per project, grade 
level, or sampled projew per set. The sample sizes, however, provided 
sufficient data for ^^^ctlng a variety of analyses dealing with the 
type and level of Impact, and for determining whether certain factors - 
such as student characteristics (e.g., grade, level, age, sex, extent of 
project participation) affected those Impact findings. 



g 

Students 1n grades 4-6 received a slightly more simplified version of 
Parts II and III than students in grades 7-12, 
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In particular, extent of participation in project activities was an 
Important variable, since pre-post school year shifts In attitude 
toward school, pride In being an Indian, and self -concept, for example, 
were then related to extent and type of program participation/ Those 
who received relatively light amounts of certain program components 
therefore became a natural comparison group (after controlling for 
student characteristics and similar factors) for those who a 
greater amount of program exposure. Having sufficient sample sizes at 
each grade-level range made such analyses extremely useful and sound. 
Having such comparisons as base! i|ps for detecting project participation 
effects was also Important to the ^tudy. 

One of the most difficult aspects of the study was the construction of 
the student sample, When the data collection plan was being developed, 
there was considerable uncertainty concerning the size of the relevant 
student population, - The school districts included in the sample were 
contacted prior to site visits, and were requested to provide estimates 
of the number of Indian students in grades 4-12 who were 1n„ schools and 
grade levels where they could be receiving ^Part A services in the . 
coming year. Based on the responses of districts, it was estimated 
that there were approximately 50,000 eligible students In grades 4-12 
in the 119 selected districts. 

It was recognized, however, that this estimate of eligible students was 
imprecise because it assumed that district adninlstrators: (1) could 
accurately predict the number of Indian students* (2) would provide 
unbiased estimates of those numbers to an evaluation contractor working 
for the federal government; and (3) knew which schools and grade levels 
would be receiving Part A services. Because of the Imprecision of the 
population estimate, 1t was decided that probability sampling would be 
done using sampling fractions rather than specific target sample 
sizes, (Thus, for example, data collectors were told to have "l/a" of 
all Indian students in a school complete questionnaires rather than 
"40" students.) In this way, even if the estimates of eligible Indian 
students were Incorrect, the probabilities of selection for all 
students would remain approximately equal and adhere to the original 
sampling design. 

The desired sample size was 15,000 students. Because it seemed 
possible that the population estimate of 50,000 eligible students was 
positively biased, the sampling fractions were adjusted upward by 15-20 

per cent to ass ure a sample of at le ast 15,000 students. During data 

collection, additional information was obtained concerning the number 
of Part A-eligible students in the selected projects. This Information 
confirmed that the Initial estimate of eligible students was too large, 
and Indicated that the actual total number of such students was closer 
to 41,000. The sampling fractions which were used thus should have 
produced a sample of slightly fewer than 15,000 students, 

The actual number of questionnaires completed In the fall data 
collection was 13,538. This number was Inflated somewhat by 20 sites, 
from which approximately 1500 more questionnaires were received than 
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were requested, based on the stipulated sampling fraction. At one 
site, for example, school district officials requested that all Indian 
students? be included when only a sample was needed. Analyses were 
subsequently performed to determine if the inclusion of these 
additional students would significantly affect the student results. 
The analyses showed no major effects, so the students were included In 
thj§ analyses. Based on the best available estimates, therefore, the 
loss rate for fall student questionnaires was In the range of 16-20 
percent. Some of that loss can bk attributed to the four of 119 sites 
from which no student data had been received. Approximately 400 
student questionnaires were expected from those sites. In addition, 
approximately 100 questionnaires were received which were unusable 
because they were Incorrectly or only minimally fined out. These 
questionnaires covered a variety of projects, schools and grades. The 
remaining data loss can be attributed to students who either: (1 ) .were 
not attending school on the day of testing or (2) were in school but 
failed to attend the survey session/ 

It is reasonab 1 e to con j ecture that the data"Toss~from the s tudeht" ™ : 
sample could have biased the results f^om the student questionnaires. 
That is, it could be hypothesized that the students who did not 
complete questionnaires had generally poorer attendance, poorer 

* academic achievement, and relatively less positive attitudes toward 
school .and themselves did the students who completed questionnaires. 
If such bias existed, however, it should not have had a major effect on 
the study results. Assuming a 15 percent loss rate, even if the 
nonrespondents were a#ull standard deviation below the respondents on 
a given measure (an unlikely amount of bias), the bias in the study 

, mean would have equaled ,15 of a standard deviation. Such differences 
would not Have changed any of the major study conclusions. 



(g) Parents of Selected Part A Participants . This data source was an 

Important one, both for supplementing and verifying student perceptions of 
benefits due to participation In project activities, and for gaining an 
(accurate sense of what benefits parents believed their children had 
received from the project. Therefore, a sample of parents was 
administered a Parent Interview Guide during the sprang data collection 
period* • • T~ ■ 



The parents who were interviewed were a stratified random subsample of the 
parents fit participating students, who were given the student 
questionnaires during the fall visits. The number of parents selected for 
interview purposes varied from site to site, approximately in proportion 
to the number of students surveyed. At each site, the list Wf names of 
selected parents and alternates was provided to the project director. 
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together with a draft letter requesting an Interview, The alternate names 
were used only if problems were encountered which made it Impossible to 
interview a parent on the primary list, Every effort was made to contact, 
schedule, and Interview ap parents on the primary list, 

The project directors assisted in locating the parents and notifying them 
of the survey prior to the^ield staff 1 ! arrival. The project director 
was given the discretion of Initiating the letter or making other types of 
contacts with parents prior to this team's arrival, A list of the 
parents* telephone numbers and addresses (with directions) was provided by 
the project directors. All parent interviews were completed while the 
team was on site. 

There was a possibility that during the spring the corresponding student 
might have either* (a)^^en absent from school, (b) dropped out of school 
yet remained a community resident, or (c) moved away with his/her family. 
Wherever possible, parents selected for the sample were interviewed during 
the spring visit to the site, regardless of the student's having fallen 
into any of these three categories. To ^ipimize the attrition of selected 
parents from the sample, every effort was made to simplify the interview 
process. (This Included using flexible scheduling of local Indian \ 
interviewers familiar with neighborhoods and the local language, / 
interviewers' traveling to homes where considered necessary, etc.) 
However, since extensive follow-up efforts were costly and beyond the 
scope of this study, parents who had moved from. the community were not 
contacted. 



Samp! ing rates ahd stratification. As iridicated abbve, to obtain a sample ^ 
of parents, a subsample of students In the student sample was selected and 
the parents of students in this subsample were interviewed. To select the 
subsample, sample students were stratified by grade level and by type of 
program component in which a student was participating. On the average, 
the goal was to contact six element«^(grades 4-6), six junior high 
(7-§), and six high school (10-12) parents, - 
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Overall , 1,54$ parents from the 115 sample projects were Interviewed, 
This Is an average of over 13 parents per project. Thus, sufficient 
parent data were available for analysis in terms of several factors that 
may have been associated with the results (grade level of student, type of 

program component, sex of student, etc, ), > . 

== =*■ j, 

J 

(h) School Principals , The principals or the designated representatives of 
the schools In the sample were Interviewed during the spring data 
collection period* The principal or the designated r^resentative such 
as the assistant principal was Interviewed by using the Principal 
Interview Guide, which focused on: f 1 ) the effects of the project in the 
school-and- (2)— the- function ^f-p^ 
with instructional and other school activities, 

The goal was to contact the principal of every school in the sample, *In 
small projects with only one sample school, only one principal was 
contacted, In larger projects, a minimum of two principals were 
interviewed. The average number of principals surveyed per project was 
four, 

* * . 

(1) Regular Classroom Teachers , Regular classroom teachers of the 

Indian/Native students selected from schools where students were surveyed 
were atfninistered the Teacher Questionnaire in the spring. At the 
elementary school level, the questionnaire was given to the regular 
classroom teachers of the Indian/Native students in the sample who had 
been in the school system for at least two yeara (i.e., then in at least 
their third year). At the junior and senior tiigh school levels, these 
questionnaires were given to the mathematics, English, and social studies 
teachers of the selected Indian/Native students who had been In the school 
system for at least two years (1 .e, , then In at least their third year), 

It was believed that the best estimate of Impact, as provided by classroom 
teachers, would come from those teachers who had worked in that school 
system for two or more years. Such teachers were thought to be able to 
assess the occurrence of certain effects over the past two yeajgs more 
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readlly, and to judge whether such trends seemed attributable to Part A 
project operations and activities. This subset of all regular classroom 
teachers instructing the selected Indian students was therefore used, 
rather than the full set of all such tochers.. 

Some school districts had too 'many schools, grades, and teachers to 
include all in the teacher interviewing process. \ This was especially true 
at the middle or secondary school levels, where more than one teacher ^ 
instructed given students, In this case, teachers were subsampled by 
choosing a random sample of regular classroom teachers. The decision to 
use subsampling methods was made on a district-by-distr1ct basis. 1 

Taking these considerations into account, about 11 teachers per project, 
or a total of 1 ,307 teachers, were contacted. This sample size made It 
possible to prepare comparisons across grade-level ranges (e.g. , 
comparisons of grades 4*6 teachers with those of higher grades on certain 
common dimensions) and other useful subgroup analyses, 

(1?- Selection Criteria . The criteria ^d procedures used by the field' 
staff for the selection of the teacher sample are described below: 

* Teachers in the sample: (1) had taught In the school district for 
at least the current and previous two school years* (2) had at 
least two eligible Part A students in their classes; and (3) had 
taught in schopls where students were asked to fill out 
questionnaires* 

ft Three randomly selected teachers at each grade level (i.e., three 
from grades 4-6, 7*9, and 1 0-1 2, for a maximum of nine per school) 
in each school where students filled out questionnaires were given 
teacher questionnaires, , 

At each elementary school, one regular classroom teacher from each 
grade 4-6 was given a questionnaire. If there was no teacher at one 
grade level who met the sampling criteria, a teacher was substituted 

from the next grade level, 

<i ■ 
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At the junior and senior high school lev§l$J( grades 7-9 and 10-12), < 
questionnaires were given to one math, lone social studies, and one 
English teacher at teach level in each school . 



Exceptions: 

•In districts where Part A-eliglble students were widely scattered 
and where fewer than ten Part A-eligible students were at a 
particular school, a questionnaire was given to the teacher in the 
school who had^the most Part A students* If no teacher had two or 
more Part A-eligible students in his or her classes, field staff did 
not distribute any Teacher Questionnaires at that school, 

, • In schools With fewer than three teachers meeting the sampling 

criteria, the questionnaire was given to the one or two teachers who 
had at least two Part A students, 

• In school districts with a K-5, 6-8, and 9-12 grade breakdown, K-5 
was considered the elementary level , 6-8 junior high, and 9-12 high 
school, In any school district with other breakdowns, field staff 
followed the criteria as closely as possible while distributing 
about the same number of questionnaires, 



(2) Procedures , To select the teachers who were asked to complete 

questionnaires, the field staff identified all teachers in the sample 
schools who met the stuty criteria stated above, Principals were 
asked by project directors to draw up such a list of their teachers 
prior to field staff arrival. If this was not done, the principal of 
each school was contacted upon arrival and the sampling criteria were 
explained, The principal was then requested to Identify the teachers 
at each grade level or in the relevant subject areas who fit the 
criteria. From this list* one teacher from each grade level or 
subject area per school was selected by using a random numbers table, 



( j ) Tutors of Indian Students , During the fall site visit all tutors of 
Indian students supported through the Part A Program were asked to 
complete questionnaires describing their characteristics (e,g, # sex, 
experience, number of students served) and characteristics of each of the 
Indian students they tutored {e,g,, grade level, proficiency in reading, 
math, and other subjects, interest in school, school conduct, amount of 



tutoring)* Project directors were also- given V package of these 
questionnaires and asked to see that they were completed' during the year ^ 
- - as new tutors or students began to participate In the Program, Project 
directors were also given a package of Post-Tutorial Follow-Up forms and 
asked .that a form be completed for each student at the time the student 
stopped receiving tutoring or at the time of the study's spring site 
vlsltl whichever came first. During the period between the fall and 
spr1ng\ site [visits, project directors were reminded several times by 
letter and telephone to have these forms completed as appropidatev In 
total, fulY data were obtained from 329 tutors and 3*526 Indian students, 

f * 
(k) Cohorts ok Post High School Students , In order to assess trends In high 
School completion and post secondary school activities* a sample of five 
Indian high school sophomores In each of the following years was selected 
from each of *the visited projects: 1970-71 , 1972-72, 1974-7S, 1976-77, 
1978-79, Indian students were Identified by school district staff and the 
students in each group were selected at random, Information about those 
selected was then gathered from school records, school staff, family, 
friends or other available sources, Several projects jfere located In 
. districts without high schools and were not Included in this aspect of the 
. study. In other districts, where there were five or fewer Indian high 
school sophomores, all Indian suphomores were Included* Overall, 2,098 
former students were included, or an average of over 18 students per 
sampled district, 

4. Selection of Public School Districts for the Alternative Resources Sample 

The population for this sample (3,177 districts) Included all eligible public 
school districts (I.e., those with ten or more American Indian or Alaska 
Native students and those located in Alaska, California, or Oklahoma) except 
tribal ly-control 1 ed schools. This section describes the method of sampling 
the part of the population of 3,177 districts which were not included in the 
survey population for the Local Project Impact Evaluation Study, 

.4 
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(a) Sample Design, To draw the sample for the Alternative Resources Study, 
four major strata were defined according to the number of American Indian 
or Alaska Native students In the LEAs. Optimum allocation of the sample 
to these four size strata was not possible, sfnce* standard deviations of 
Important parameters within straia were unknown. However, general 
experience with sampling suggested allocating the sample to the size 
strata In proportion to the total number of students In each, : w1th an 
upward adjustment in the size of the sample from the two strata having the 
smallest number of students, That Is, compared to the optimum allocation, 
*n allocation of the sample in proportion to the number of students was 
^^Red to undersample the first two strata (those with LEAs having the 
Smallest number of students), particularly the first onte, 

^Within each of the four size strata, the projects were arranged by state 
within the 12 geocultural regions, A systematic random sample of 
districts was selected with equal probability within each of the firs^ 
three size strata. The last, or fourth, stratum (LEAs with a large number 
of students) was treated as a special case, because the allocation called 
for a sample of 12 LEAs Jrom a total of 14 within that stratum, The ten 
largest were selected with probability equal to one. Two of the four 
smallest In this stratum were in California, and, two were in Oklahoma. 
One LEA was selected at random from the £wo in California, and one from 
the two In Oklahoma, 

In suninary, 108 districts were chosen, 98 by using systematic random 
sampling with equan probabilities of selection within strata, and ten with 
probability of certainty. 

The selected projWfs were administered either: (1) the Excluded Projects 
Telephone Survey Questionnaire or (2) the Non-Funded Districts Telephone 
Survey Questionnaire** Each version of the form was compatible with the 
ether, and wljth similar data gathered from the 115 sites. Three types of 
information were collected by using each version: 

o 

• The total amount of federal education funds expended by local school 
districts on Indian/Native students jn grades K-12; * 
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• The number of Indian/Native students served by these funds; and 
m The types of jpecial services these students received. 

In each df strict, the relevant^ administrator was contacted for costs and 
services/activities Information, Typical ly, this was the district's 
federal program coordinator, Telephone interviewing was chosen as the 
most cost-ef f ecti ve way to contact these projects. Site visits would have 
been needlessly expensive given the data required, and a mail survey was 
likely to produce a fairly low response rate from districts that did not 
receive Part A funds. However/ by using telephone interviewing, a 
response rate of at least 901 was expected, With call -backs made as 
necessary,, this response rate was achieved, ^< 

S. Weighting of Data from the Impact Sample of 115 Projects % 

Whether data of a particular kind needed to be weighted was related primarily 
to whether sampling within projects was involved. As_ explained above, the 
sample projects were not selected with equal prpbabllitfes and was clear 
that "project data" needed to be weighted, owing to the wide range In the 
probabilities of selection, "Project data 11 refers to data where there is no 
sampling within projects, such as data obtained from the project director 
about his own. characteristics or about the project as a whole. In other 
words, the probabilities involved in the selection of projects are applicable 
to "prqject data." "Within project data" refers to data where sampling within 
projects was involved, the prime example being student data obtained from' 
samples of students within the selected projects. Project data and^within 
project data will be discussed separately. 

Of the ten sources of data presented In the section "Sampling of Respondents »" 
the first five did not involve sampling within projects, with the exception of 
the sampling of parent committee members. The last five all Involved sampling 
within projects, except for some data in a few of the smallest projects where 
all eligible respondents were Included in the survey. 
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Project data fall Into one of two major categories with regard to/weight/ng. 
The first consists of data where use of reciprocal j of probabilities of 
selection as weights Is appropriate with regard to both random sampling error 
and bias. Such data (e.g. project budgets) are quantitative data, A Variable 
which can range In value from zero (or a small amount) to an amount that Is 
V related to the size of project generally belongs In this category. That Is, 
for such a variable, the variance among large projects is much greater than 
the variance among small projects, 

Weights for data in the first category have been referred to in this - stucjy as 
the Wj set of weights J° These weights are the reciprocals of the ' 
probabilities which the* projects had of being selected, multiplied by a ■ 
constant that makes the sum of the weights equal to 865 s which Is the total 
number of projects in the survey population* In a probability or sampling 
sense, the weights provide unbiased estimates, Also, for variables ffi 
the first category, the U 1 weights are most appropriate In the interest, of 
minimizing the standard error of the estimates, or, conversely, providing 
optimal precision, ^ 

Most of the project data are in the second category. This category includes 
variables where the variance of the variable has little if any relation to 

size of project. That is, Its variance among large projects is roughly the 

— -~ .* 

same as Its variance among small projects, Attribute or "yes -no" type of data, 
are included in this category, Examples are types of programs sponsored by a 
project or certain characteristics of a project director, Were the f 
weights applied to data in this category, the standard errors of estimates 
would be too large. Not weighting such data would provide estimates with much 
lower sampling error; however, the potential for bias in the estimates is too 

great to Ignore, * 

* , ■ j 



10 Found on computer data files as the variable WT1 , 
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Thus, a second set of weights, referred to as W 2 . was prepared with an 
objective of minimizing the mean square errors of the estimates; Mean square 
error Is a combined measure of bias and random sampling error,. The Wg 
^weights were formed by adding a constant to each W 1 weight and scaling them 

so that the weights would add to 865. The constant was chosen so that the 
f maximum W 2 weight within a region would not Be more than about four times 
larger than the smallest weight within the region, 

Data from parent committees were regarded as project data* even though some 
data were obtained from a sample of Individual members of a committee, In 
this case, the committee, rather than a committee member, was treated as the 
unit of analysis* In other words, if data from all parent committees in the 
population of 865 projects had been obtained, It was' assumed that committees 
would be the tabulation units, All committees would then have the same 
weight, regardless of size, Under this concept of the parameters 1 being 
estimated and considering the fnnd of data Involved, the W 2 weights were 
applied as follows, If there were data from n members of a coninittee s the 
weight for each of the n was W 2 /n, where Wg Is the weight for the project 
Involved. Then, the sum of the weights for the n members is equal to Wg and 
the sum of all weights totals 865, the total number of projects, rather than a 
population total number of coranlttee members. 

Tribal /committee leader data consisted of two or three respondents from some 
projects, although there was a relatively small number of projects with more 
than one tribal leader respondent. As in the case of committee members , ^ 
weights were assigned equally to all respondents from the same project, so 
that the sum of the weights for respondents in the project was equal to the 
projects' Wg weight, . " 



n Found on comp liter data files as the variable WT2. 
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Adjustment for non-response* When data for one or more of the IIS projects 
were missing, the weights (either the or Wg weights, whichever applied) 
were adjusted within regions so the total weight for a region remained 3 
unchanged. For example, suppose there were 1 1 sample projects in. a region and 
data for one project was missing* Assume the sum of the weights for all 11 4 
projects is 95,0 and the sum of the weights for the ten projects with data 'is 
87.6. The ten weights would be adjusted upward by the factor 95.0/87.6 - 
1,08* In case a region was small (say, five or six projects) and data were 
missing for one or more projects in it, the region would have been combined 
with another region so the adjustment factor would not be large and the 
adjustment would be spread over several projects. ' \. : 

t _.; _ " \ :y- : ' : 

Wi thin-project data . A major part of the w1 thin-project dat^ consisted of 
data from students, and it will be discussed first, As explained earlier, . •-. 
sampling fractions for, sampling students within projects were calculate^ to 
give every student. In the population of 865 projects, an equal chance of 
being In the sample. For various practical reasons, this was not completely 
accomplished, The question for consideration was then whether the departures 
from equal probability were sufficient so that weighting would be advisable, ;< 
To answer that question, a set of weights (applicable to student data 
collected In the spring and based on reciprocals of sampling fractions) was 
calculated and studied. Also, some data were weighted and the results were 
compared with corresponding results when no weighting was done, A conclusion 
was reached that differences between weighted and unweighted results would be 
trivial or unimportant. Moreover, when analytical processes are complicated, % 
the weighting of Individual -data adds much to the burden of processing. Thus, 
a decision was reached to treat the student data as self-weighted, 

Since the numbers of parents in the sample from project to project were 
approximately in prtyiortion to numbers of students in the sample. It follows 
that weighting parent data was unnecessary. 

Two other sources of wi thin-project data were principals and school teachers. 
These data were also approxlrpately self-weighted and were not weighted because 
weighting would not have Improved the results by an appreciable amount, if at 

an. , - 8x 
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For the post-high school student sample,* an upper limit of 25 cases per 
project was placed on the sample/ This meant that large projects were 
somewhat underrepresented, ev^n though large projects had higher probabilities 
if being 1n the sample. The weight of each project was calculated and" 
examined. After reviewing the pattern of these weights* a straight-forward 
set of alternate weights was regarded^ as adequate, Data for sample students 
in the 18 largest projects wer4 therefore given. a weight of 2*0 and data for 
all other students received a weight of 1 .0. 

6. Weighting of Data from the Sample for the Alternative Resources Study V 

For -sampling purposes, there were four subpopulations of the population .to 
which the Alternative Resources Study applied, The four subpopulations are 
shown in Table^ 3-10, The first subpopulation Is the population of 866 
districts that was defined for the Impact Evaluation Study v The set of 
4 weights described above is applicable to the sample data from, this 
subpopu]ation . In the second subpopulation, equal probabilities of selection 
jyere used- within strata and the stratum weights shown in Table 3-10 expand the 
sample from this subpopulation to estimates of totals for this subpopulation, 

The other two subpopulations are small and the weights shown In Table 3*10 
also 4 expand sample data to estimates of totals, for these two subpopulations, 
ThCis, estimates of totals for each of the four subpopulations were added 
together to obtain estimates for the entire population. 

7 , Accuracy of Estimates " " 

This section contains the standard errors for those Part A Program 
characteristics' which were selected as 'vbeing particularly Interesting to 
policymakers and planners. Some guidelines for interpreting these standard 
errors and how they were computed are also Included/ 



12 Found on computer data files as the variable WT3, 
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PATTERNS OF WEIGHTS APPLIED TO SUBPQPULATIQNS OF THE ALTERNATIVE 

RESOURCES SUBSTUDY 



* 

Subpopullflon 


Number of 
Districts 
in Population 
Having Indian 
Students 


Number of 
Districts. 

inciyoea 

In Sample 


Assigned Weight 


a, Fyndad projects 
for three years 
with more than 
30 Indian students 


865 


119 




b. Non-funded districts 
(N-2179; 449 without 


1730 


100 


By stratum (number of- 
Indian students): 


Indian students) f 855 
Four Strata < 719 
) 144 
12 


20 
43 
25 

, I 2 


4er75— 
16.72 

5,76 

1.00 


c. Projects funded 1n s 
1979 jHid 1980, and 
not An 7981 


81 


5 

* - 

f 
\ 


Four with weight=20.0; * 
one with welght-1,0 
(which was. sampled with 
probability equal to T 
due to its large size) 


d. Projects funded with 
fewer than 30 Indian 
students 


52 


3 

f 


17,33 


*An unbiased weighting* factor; see. the project data weighting discussion for a 
fuller explanation, "V 
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Two genepal types of errors are coninonly recognized 1n deriving summary 
statistics or estimates based on a sample survey sampling and nonsampl ing 
errors* Sampl ing errors occur because the obtained da'ta are based on a 
probability sample rather than the entire population, Nonsampl ing errors 
arise from many sources other than sampling, and represent an entire area of 
concern in themselves. They can arise from any of the following factors: 
Inability to obtairt information about all cpses In the sample; definitional 
difficulties which may vary across local projects or respondents* how 
questions are interpreted; respondents' inability or unwillingness to provide 
accurate and correct information; and^a wide range of other measurement, 
pfocessihg ? and responding errors, H ft 

The national -level estimates provided here are obtained from sample data, and 
jtherefore^vwny^^mewfaat^ 



obtained had a complete survey or a census yielding \00% response been 
conducted using the same data collection forms, procedures, and Instructions, 
Furthermore , any sample Is only one of a large number of possible samples (of 
the same size ) that could have been selected by tis/ing the same sampling design 
and universe of projects or other types of sampling units. Estimates derived 
from these different samples will generally differ from each other, Such a 
difference between a sample estimate and the average of all possible samples 
(drawn from the same universe) is called a sampling deviation, In turn, the 
standard or sampling error of a survey estimate Is a measure of the variation 
among the estimates from all possible samples. It therefore Is a measure of ^ 
precision with which an estimate from a particular sample approximates the 
average result of all possible samples* 

In general , the sampl ing procedures and sample sizes used In this study were * 

selected to minimize tr*ror to the extent possible wrftiln reasonable costs, 

recognizing the finite resources available, In additions while the standard 
* ' ■ > . v 

error partially measures the effect of nonsampl ing errors, It does not measure 

systematic* biases in the data. Bias (or mi srepresentati veness } is the 

difference, averaged over all possible samples, between the estimate and the 

.true-value. ^With_these -factors- iii_mind»-the- overall— accur^ 

result depends on both: (a) the sampling add nonsampl ing errors, measured by 

the standard error and (b) the bias, and other types of nonsampl ing error, not 

measured by the standard error, 
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The sample estimate and an estimate of its standard error permits the 

development of interval estimates with prescribed confidence that the Interval 

Includes the average result of all possible samples. For example, one of the 

most frequently cited confidence interval sizes Is the 95$ confidence 

interval. Conceptually, this means that If all possible samples were 

selected, If 4ach was surveyed under essentially tht same conditions, and an 

estimate and Its estimated standard error were calculated from each sample, 

approximately 95 percent of the Intervals from 2 standard errors below the 

estimate to 2 standard errors above the estimate would Include the average 

value of all possible samples* (An Interval from 2 standard errors below the, 

estimate and 2 standard errors above the estimate is called a "95-percent 

13 

confidence interval"; see Gonzalez, Ogue, Shapiro, and Tepping, 1975. J 



Sampling error . As only one primary sampling unit (a project) was selected 

from a stratum. It is necessary to use approximation methods to estimate 

J 

sampling variances. Techniques known as collapsed stratum and ultimate 
cluster were used. A collapsed (or pseudo) stratum is a combination of two or 
more of the original strata used in the selection of projects. The objective 
is to combine similar strata and total the combinations (pseudo strata) as 
though they were the strata used in the design and selection of the sample. 
An "ultimate cluster" 1s the sample of units selected from a primary sampling 
unit, which in our case is a project. 

Let n h equal the number of ultimate clusters In pseudo stratum n. Within 
each stratum, n h estimates were prepared, one from the data for each 
ultimate cluster. Each of these estimates was an estimate of a total, an 
average, or a percentage for the pseudo stratum as a whole. Let x^^ 
represent the estimate based on data from the 1th cluster In stratum n. For 
purposes of estimating sampling variance, the values of x hi - are treated as a 
stratified random sample with proportional representation from each pseudo 



13 Gonzalez, M. , Orgue, J., Shapiro, G. & Tepping, 6. Standards for 

discussion, and presentation of errors in survey and census data. Journal of 
» the American Statistical Association , 1975, 70, Part II. 
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stratum. That ii; the well known variance formula for stratified random 
sampling with proportional representation from strata was appl led. The 
estimates of variance tend to overestimate the actual variances, which are 
probably of very HttTe importance for practical purposes. 

As discussed above, sorae/<Jata did not involve sampling within projects. In 
this case, the same approach to estimating sampling variance is applicable. 
The value of the variable' for a project multiplied by a appropriate expansion 
factor or weight becomes , an estimate for the entire pseudo stratum in 
question, ff 

The_ estimates of the_ _s tan da rd erro r fo 'r sej ec te d _e_s_ti^ _P r?sen te d_ i n^ 

Table 3-11 - In general , the largest standard errors are for estimates that 

* are based on data from district administrators, project directors, parent 
committee chairpersons, and other data where the sample size or number of 

—units of observation is 115. Estimates based on data from students, teachers, 
parents, and other sources that involve much larger numbers of units of 
observations have the smallest standard error, Tab! 1f}3-1 1 shows that for 
estimates of percentages, the standard errors are llffthe neighborhood of ZS 
percentage points where the reporting units are students, teachers, etc. On 
the other hand, where the percentage estimates are percentages of projects 
having particular characteristics (e.g., percentage of projects having various 
program components) the, standard errors are on the^order of 5 or 6 percentage 
points. 

8. Procedures Used for Dealing With Item NonVesponse 



ir sell 



The weighting approaches described above have dealt with the situation where 
entire questionnaires were completed and needed to be properly (a) weighted to 
reflect a larger population or (b) weighted* then adjusted to compensate for 
respondent non-response, enroute to making national -level estimates and 
_correla_t1.ona_l_use of thje oMaxned data. 

, v - 
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TABLE 3-11 

ESTIMATED STANDARD ERRORS FOR SELECTED ESTIMATES 





Variable 
No. 


Characteristics of Variables 


Estimate 


Estimated 
Standard 
Error 




\ 


1 


Project Director Characteristics 

m Indian « 
% Male 

i Having a degree 
Ave. hours worked/wk 


55,0 
83,1 
19,0 


3.£ 

7,8 
7,8 
1,6 






2 


Program Components 

% With counseling 

% With home-school coordination 


48 

38 


6,0 
5.8 








* w i tn parental cost component 
' % With cultural activities 
% With tutorial program 


_ — vo~' ™ 

64 
80 


r- c 

3.9 
5,4 

3.9 






*3 


Hours Tutored Per Week % 


4,7 


0.2 






4 

5 


Parent Corranlttee Chairperson 

% Male , 
Ave, number of years on coranittee 

r 

Indian Students * 

% Leaving before completing H*S, 
* witn education oeyond n-5. 


42 

2,3 

19 

ill 


8.0 
0,1 

1.6 






6 


Project Staff 
% Native 
% Female 

% Employed full time 


71 
84 
* 81 


3,6 
2,9 
2,7 






7 


parent Corranlttee Members 
% Indian 
j % Male 

% Parents 
% Teachers 

% Satisfied with project 


87 
30 
71 
12 

89 


1.7 

2.7 
2.0 
1.7 
2,1 






8 


Indian Parents who think 

Project has helped their children 
to get better grades 
Project has helped them personally 


% 68 

% 57 ; 


2.1 
2.2 
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TABLE 3-11 (Continued) 





Variable 
No, 


• 

Characteristics of Variables 




Estimate 


Estimated 
Standard 
Error 




9 


School Principals 




89 
89 








) Satisfied wi th qual rty of project 
Believe project is valuable 


% 
% 


1.9 
1.9 




10 


Teachers have made changes 

In curriculum material 
t In teaching approach 

* 


% 

. % 


41 


Z,U 
2.0 






Teachers believe 

More parents are Involved 
Project is benefitting students 


% 
% 


60 

, 78 


2.8 
2.4 




n 


Student Achievement Test Scores 

Reading * ave. score (in T-score units) 
Mathematics - av# 9 score 
(in T-score units) 


47.4 
47,8 


.55 
, -51 




12 


Average Days Attended School s 1 980 




162.4 . 


,90 



J 

4 

k 
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In addition* there ware Instances of Item non-response which occurred across 
the various respondents and types of data collection instruments^ used in the 
study, A review of, all Items having such non-response was made during the 
data management phase of the study. No consistent or systematic pattern of 
item non-response seemed to exist and Instead was apparently scattered, , No 

attempt, therefore* was made to Impute data values for those Items to the 

projects, and to the various respondents which were Included in, the sample and 
kho supplied less than 100S completion of the Information sought from them. 
Although several alternative methods exist In the field for making such 
Imputation efforts (such as expanding th# weights assigned to respondents on 
given items to reflect the proportion of non-respondents } , It was felt that 
any such method would be relatively imprecise. Overall , item non-respoinse 
represented a relatively small fraction of the data. Ar such* it was judged 
that adjustments would not contribute materially to changing the estimates 
reported or to Improving the precision of the findings presented. 

With that as a guideline* it algo was felt that statistical tables and the 
discussion of the results contained in them should clearly present the number 
of unweighted and weighted cases on which findings are based. Similarly* 
findings presented on a subset of all respondents indicate the size of that 
subset* for the sake of making the findings as lucid as possible. To make 
tables more readable, when all variables within the given table are based on 
only slightly varying numbers of respondents (i.e.* within 10 percent of the 
maximum number) the range of cases 1s provided as a footnote to the table. 
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CHAPTER 4; IMPLEMENTATION OF TOE STUDY DESIGN AND .METHODS 



Summa ry of Tasks 

During the initial stages of the stucty, the goal was to acquire a thorough 
understanding of the Part A program at all levels — ' congressional , 
Department of Education, and district/project level, To gain this 
understanding/ an eval usability assessment was conducted during the first 
several months of the stu^y, > , 

As the study began, there was much discussion of the limitations and 
opportunities for conducting such an impact evaluation^ These became more 
clear as the evaluabll 1ty assessment (the Preliminary and Exploratory phase I 
progressed* The approach from the onset was to maintain a flexibility of 
design and operation that would be open and able to capture the diverse 
program and Its human contexts. One large concern was to ensure and maintain 
sensitivity to, and Involvement of, the program participants and ©the^J 
potential users of the, evaluation results* Again, this required maintaining 
a sensitivity and responsiveness to the suggestions, observations* and 
opinions of individuals and groups at all levels of Part A Involvement. 



~T6 describe this further," ffi "were to acquire a thorough- 

knowledge of the Part A Program at all levels within its political , 
governmental, social, economic, and local contexts, Prior to any evaluation 
design considerations, the evaluaMlity assessment required the contractor to' 
examine the legislation, regulations, congressional hearings, budget and 
program justifications, program history, financial history, the current 
program and its status, Part A records and applications, and the National 
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Advisory Committee on Indian Education (NACIE) Annual Reports. In other 
words, germane documents Included any written material that could be drawn 
from the public and from governmental records Which were worthy of providing 
insights Into the Program's purpose, characteristics, history and operations, 

. 01 scMssi ons and 1 n te rv lews we th 
the appropriation and authorization committees responsible for the Indian 
Education Act, as well as Department of Education and Office of Indian 
Education Program officials. Prior administrators of the- Indian Education 
Program were also Interviewed, Visits were made to local projects during the 
exploratory phase of the study. (A total of 20 projects In various settings, 

representing a diverse cros^sectl on, were visited.) 

J _ 

In addition, a Technical Advisory Panel was established, whose function was 
to provide guidance and direction throughout the stu<fy. This panel was made 
up of individuals from the Department of Education, the Indian Education » 
Program office. Bureau of Indian Affairs, the NACIE Board, and Indian 
educators representing special fields of expertise, (Seven of the eleven 
members were Indiag. ) Over 20 Indian consultants were also hi red. and 
contributed to the design of the initial study and the data collection 
instruments, * 

In summary, the purpose of this phase of the study wastthe acquisition of 
information and knowledge about the Part A Program from all possible 
perspectives and sources. This was accomplished by: - 



Research and assessments of the legislation, regulations, policies, and 
administration involving the Part A Program and of current and past 
research studies and evaluations of, and about, Indian education, 

Reviews o^IEP records, files. Part A-funded appl/cations, NACIE Annual 
Reports, congressional hearings, and Office of Indian Education budget 
justifications to the Congress, f 

Interviews with congressional staff, Department of Education officials, and 
the~Indi an Education" Program~offtcTal^ aird staffs — — — -'— 
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• Press rel eases to Inform Indian/Native people about the evaluation 
projec.t and to solicit recommendations and comments* 

• Consultations with education specialists with emphasis upon 
Indian/Native educators with diverse expertise from different tribes 
and regions, 

• Technical Advisory Panel meetings to provide periodic review and 
approval of the work as it progressed* 

• Information and data collected on the operations and achilnl strati on of 
the Office of Indian Education for the period of 1976-80; the 1,094 
funded Part A projects for 1980-81; and the 1 ,080 funded Part A 
projects for 1981-82. - 

• Preliminary site visits to nine Part A projects to obtain a first-hand 
view of some diverse projects In different areas of the country. 

• Field tests of the Interview forms, questionnaires, and field data 
collection procedures at 13 Part A projects in di f f erent parts of the 
country, representing diverse program activities, Indian student 
population size, and locations (i.e., rural, urban, reservation, Alaska 

^ Native village}. 

Development and Implementation - 

During the developmental and design phases, consultations took place at the 
V federal level with staff of relevant congressional committees; the Office of 
Indian Education; the Office of Planning and Budget in USED; the Bureau of 
Indian Affairs; and the Bureau of the Census. There were also consultations 
with staff of local projects and their school districts, Indian/Native 
parents and students, representatives of national Indian organizations, and 



experts on Indian education and evaluation design from across the U.S. Study 
materials, including the completed instrumentation package § were .provided to 
Dr. Ron Torgeson of the North Dakota State Education Agency, as the official 
liaison of the Council of Chief State School Officers (Committee on 

Evaluation and Information Systems), prior to the visits to school districts, 

— S • 

Study staff worked with a Technical Advisory Panel of nine members, seven of 
whom were Indians, with recognized expertise In various areas of Indian 
education, evaluation, and testing and measurement. Including two who 



♦92 



detilopmint associates, INC, 



ERIC 



-80* 



represented the National Advisory Council- on Indian Education (NACIE). The* 
Panel advised the study* staff on matters dealing with the relationship of the 
study to previous research efforts, appropriate policy questions to address, 
and matters of Instrument design, data collection, and analysis procedures. 
The names and affiliations of Panel members are listed In Appendix 2, 



4 ** . - 

fc A large number of Individuals with expertise In Indian education, evaluation 

design, testing and measurement, data analysis, and field data collection 

were also consulted at various points to assist in developing the design and 

instrumentation or In reviewing that which was developed. The names and 

affiliations of these Individuals appear at the front of this document. 



1. Instrument Development * - 

One of the most critical aspects of a study 1s the development of 
Instruments which sufficiently cover the whole range of required and 
desirable data and information. Based upon the eight major study 
questions, preliminary Instruments were developed which were addressed to 
specific respondents and data/information sources. 

In developing the survey Instruments for this study, systematic attempts 
were made to use directly or to adapt other, already proven, 
questionnaires developed in various relevant studies. ^These included data 
collection instruments (or topic categories) used by: (a) Communications 
Technology Corporation (1978), dealing with Indian education; (b) System 
Development Corporation, used In the Title I Sustaining Effects Study 
(1978) and the Parental Involvement Study (1981); (c) Research Triangle 
Institute (1975), dealing with post-secondary student experiences, in the 
National Longitudinal Study of the High School Class of 1972 ; (d) similar 
work performed by Rand Corporation (1980) dealing with self-concept arid 
aspirations, as well as well-regarded investigations of self -concept by 
Rosenberg (1965); and (e) other topic categories in the area of attitude* 
- toward schooT by-the state^ of Pennsylvania Department^- Education - |- 
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(1980), Other studies by Abdel-Mawgood and Hitch (1972), Erickson et al , 
(1979) and SRI International (1981) have also been used where appropriate J # 

Indian educators representing many tribes In various parts of the U.S, were 
Identified and selected for their diverse kinds of experience and expertise 
and asked to help In the development and refinement of these Instruments* The 
draft Instruments initially developed by the study staff were mailed to these 
Individuals prior to field testing, Upon review of the returned comments, 
revisions were made and field tests at various projects were scheduled. Also 
solicited and used were the comments and suggestions of the members of the 
study Technical Advisory Panel (see Appendix 2) throughout the development of 
the instruments and data collection procedures, 



1 References cited above are the following: 

(a) Communications^ Technology Corporation, op cit . 

(b) System Development Corporation, The measures and variables used in the 
sustaining effects study (Report #9) ; Study of parental involvement in four 
federal education programs (see bescriptive Framework, Working Paper #2), 

* (c) Research Triangle Institute, The national longitudinal study of the high 
school class of 1972 , Symposium presentation at the American Educational . 

Research Association Annual Meeting, 197S, . 

* * ■ ..." 

(d) Rand Corporation, Effects of postsecondary experiences on aspirations, 

attitudes , and sel f -conceptions , 1 980, - 

K 

(e) Rosenberg, H, Spclety'and the adolescent self-Image , Princeton University 
Press, 1965, 

(f) Pennsylvania Department of Educattep, Division of Educational Quality 
Assessment* Manual for Interpreting Secondary School Reports , 1980 (Also, 
Manuals for ETementary and Intermediate School Reports. ) ~ 

(g) ^Abdel-Mawg0od t M.E, 4 Hatch, 6. The sel f -concept of academic -ability among 
* American Indian st udents , A Research Report. Central Washington State , 

College, 19/2. 



(h) Erickson, E, , MacDonal d/ L, { Manley-Casimir, M, A Busk, P,, Characteristics 

.... and-ralationships in pub.llc_ and independent schools: COFIS baseline survey 

interim report , San Francisco: Center for Research on Private Education, 
University of San Francisco; and Vancouver, British Columbia: EdufiaA+oHrT - 
Research Institute of British Columbia, 1979, 

(1) SRI International , j Teabher corps evaluation study * Menlo Park, CA, J 981. 



: =-^==^=SS=^ DETILOPMINT ASSOCIATES, ma. 



er|c 



The Instruments were field tested In three cycles, with several Part A 
projects visited during each cycle* The Instruments and procedures were 
modified following each field testing cycle* Thus, additional modifications 
and refinements were made^based upon the results of administering the forms, 
as well as from critical reviews by project and school district staff at each 
site visited. 



A diverse group of projects was selected and visited, representing a variety 
of activity orientations, project sizes and locations (i.e., rural, urban, 
village, reservation) * The Part A projects fn the following communities were 
visited* \ 



Grand Rapids, Michigan 
Cass Lake, Minnesota 
Arlee, Mori tana 
Dillingham, Alaska 
Missoula, Montana 
Romany Montana 



Jemei Springs, New Mexico 
Wisconsin Dells, Wisconsin 
Ft, Yates, North Dakota 
Harrold, South Dakota 
Tulsa, Oklahoma 
Ft, Covington, New York 



After the final round of field testing, the revised forms were again sent to 
the study's Indian consultants (see Appendix 3} for a final review 
comments. 



Specifically, a total of 33 discrete Instruments were developed for the 
study* (See Table 4-1 for their Ijfstlng and the number of respondents to 
each,) During the fall data collecti^ forms were 

structured int&rviews; two were sel ^administered questionnaires; and four 
were forms covering recorded. project and district descriptive data and files, 

and records data. For the spring data collection, three of the 24 forms were 

■\ * . * 

structured interviews; nine were self-administered questionnaires; ten were 
forms for recording. project, district, and student related data, including 
"data on pro j^t~and students form used for ~ — 

telephone Interviews during the Alternative: Resources Study; and one form was 
based on composite indices developed from other fprms/ * 1 . ■ . * 
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TABLE 4-1 

... " ; 



INpIAN EDUCATION IMPACT EVALUATION STUDY INSTRUMENTS 



INSTRUMENTS , 

FALL INSTRUMENTS "" 

• ; Respondents 

LEA Adml ni strator Interview Guide 154 
Project Director/Coordinator Interview Guide 114 
Parent Committee Chairperson Interview Guide 114 



Project Overview Questionnaire 114 

Program Components 

I, General Project Information „ 107 , 

IT. Cultural Activities Component 74 * 

III. Counseling Program Component 58 

IV. Home-School Coordination Component . ** 51 

V. Parental Cost Component , 27 

VI. Tutorial /Special -Academic Program Component 90 

Student Questionnaire, Parts I and II, Grades Ipf 5393 



Student Questionnaire, Parts I-IV, Grades' 7-12 ,8145 



-4 



Post High School Student Follow-up Survey 2098 ; A 

Document Review Guide .' 115 



/ Student' Attendance Section (of Document Review, Form) 8376 e students 



" SPRING INSTRUMENTS 

LEA Administrator Interview Guide 109 

% . Project Di retftor/Coordi nator Questionnaire ; . 108 

Tutor Character! sties . ' ]*tk' 329 



Characteristics .of Tutored 'Students /Post Tutorial 3528 ' 

_l . Pro je ct Staff Questionnaire ' y * ^413 . 



✓ . Parent Committee Questionnaire v * ' 290 'f* 

Parent Interview Guide ' ,: 1546 • < 

■' t > — •• ■ >.■ v i m a : u ^ — ' • 
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TABLE 4-1 (Continued) 



Student Questionnaire* Grade? ^4-6 % ■ 

Stu^j^t Questionnaire* Grades 7-1 2 V ^ * 
Principal Questionnaire - ' m ' 

* ■-. _ r ^ j _ ^ 

Teacher Questionnaire 

Indian Community/Tribal Leader Interview Guide 
Student Information Form (Reading & Math Test Scores) 



5201" 



7369 

/" " ; 450 



1307 , /■ ' -V 

102 ' 1'. 

6425VstuStits 
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District Test Scores Form * 
Total Public School Enrollment Levels by Grade «nd Year 106 



LL. 



105 
104 



District Indian Enrollment Levels Form . . 

Field Staff Summary Form (not ADP -processed) 
Field Staff Summary Form Abstract (Part of Section C only) 104 ■' . , 
Field Staff Summary Form (Items 5A-9L) 104 . *\ \ 

Equal Employment Abstract Form , 1 



101 



Alternate Resources Form (for nw-Part A-funde 
projects) N 
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Alternate Resources Form*( Part A-funded projects),- 19 

Cultural Overview Summary Form (rescoring of Project i ■ '„ • 

Overview Cultural Section data) • .74 

School Climate Indices Scoring Form 115 



2, Field Staff Identification, Training, and Data Collection Procedures 

To perform the data collection at the selected sites, American Indian arid 
Alaska Native ffeld-staff were identified and, selected during July through 
.September, 1981. To the extent possible, Individuals were selected who . 
* resided In the areas In which they were to make site visits and collect .data. 
(Familiarity with the area, the districts, and the people wit^ whom they woul 
be working was 'considered highly advantageous. ) 

All field staff were brought together for two^week-long data collection 

training sessions. The first session took place' In early October. 1981 , one 

" ay % ' • 
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week prior to the fall data collection period. The second session took pjace 
.^uring^ |he second /week In March 1982, one week prior to the ^spring data / 
collection period. Both sessions took place at the University of Oklahoma. in 
* Norman, courtesy of the American Indian Institute. w . J 

* m V r ' * .■ .. ; x ■ 

The data ^collectors were trained thoroughly In both qualitative .and quantita- 
tive data' col l^c^ion concepts, techniques, and procedures. Jhe training 3 
during .each seifsion dealt with the specific Instruments and data collection 
requirements for each data collection period. The training also Included 
procedures for onsite protocols, scheduling, Interviewing techniques, and 
qualitative inquiry techniques, to name a few, The procedures and routines 
used by the field staff wer*e designed to enhance consistency and objectivity 
as well as to minimize deviations* of procedure. 

- ■ ■ • < •' 

t . . " '. 

The bnslte visits ranged from two days to a maximum of six days per^ site. The 

fall data pol lection Involved fewer days per site than the spring data 

collection, which included parent interviews as well fljK'he qualitative 

inquiry of selected sites,- Efforts were ^1 so made durflg thf.spr/ng visits to 

follow-up on missing da tat f^om the fall, ^ ■ ~ ~ 

Local assistants were 'hired by ' the team leader at^ those sites where it was 
deemed necessary because of the heavy data collectfbn burden. These 

-individuals primarily collected file and records data, In some instances, 
they also served as interpreters during the parent Interviews at those 
locations where parents spoke their Native languages. In many of the sites, 
\project tpd district staff were volunteered to assist In the scheduling and 
actual collection of data. The cooperation and assistance of tfte districted 
project staff was extremely high In all but a very few instance^. * In fact, 
:Without the assistance of the locally]-hired assistants and, particularly, the 
project and district staff, the data collection at many of the sites could not 
have been completed, Also, during both data collection periods. Development 

^Associates' project staff, including the Project Director and Assistant 

Project Directors, visted a number of sites and assisted In the data 1 , 

r t 

collection: ■ ■ ■» 
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The field data collection staff were responsible for the collection of all 
data requfrements at 116 Part A projects in 29 states (see Appendix 3 for the 
14 st of sites). Specifically, data were collected from the following groups 
of respondents: local school administrators, -project directors/coordinators, 
project staff* parent committee members, school principals, and teachers in 
those schools where, students were surveyed, Indian/Native tribal or community 
leaders, participating students, and their parents. The structured or 
quantitative data collection Involved the adnlni strati on of a set of 
seTf -administered questionnaires (project director, project staff, principals, 
teachers, parent committee members), interview guides (parents, Indian/Native 
tribal or community leaders, district representative), and document review 
forms to a representative sample of these groups of respondents. 

The data collection was dlv.1 decl Into two time periods 1n order to minimize and 
spread out the burden, as well as to provide the opportunity to collect pre- 
and post-data from the Indian students in the survey, Questionnaires were 
administered or ihterviews were conducted with the following respondents 
during the fall data collection period (October through early December, ^ 
1981): district representatives, project directors, parent committee . 
chairpersons, and students, At this time, descriptive and demographic d*t3 
.were collected regarding the characteristics of the projects themselves and ^ 
1 tire school districts, Durirtg the spring data collection period (March through 
May, 1982), questionnaires and Interviews were acfrrinistered to the following 
respondents: the 1 same district representatives Interviewed in the fall , * 
project directors,^ project staff, parent corranittee officers, parents, 
students, teachers, principals, and tribal leaders. 

In addition to interviews^ with selected respondents, data from district, 
school, and project records were collected regarding Indian student 
attendance, achievement scores, and enrollment levels of those Indian students 
surveyed, Data We collected relative to the ext&nt of the surveyed i 
students 1 participation In the Part A Program for the current and previous 
school years/as well as their participation In the subsidized luncb program 
(the latter as a measure of socioeconomic status), Data on tutored, students 
were collected in thole projects that had a t^orial program, and a thorough 
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follow-up of missing fall data wal also dondCicted "during- the spring data . ... *. 
collection period* * ¥ \ * ' s , 

- * ...... s , , , • «' , . " . * 

Also poll acted during the spring were: /* * '- * 

t A Field Staff Summary form completed by a member of thetdata collections 
team at each site visited; f 

• A qualitative Inquiry at 31 selected sites, completed by a member of the 
data collection team; r ' 

» *■ * □ * 

• A School District trriployment Sunmary form for >i sting staffing and" 
employment data on ihk districts in the sample f completed by a study team 
staff member; - , , 

■ > ■ , . 
t The Alternative Resources Study of 106 public school districts eligible 
for but not receiving Part A monies, conducted by a Devel opment Associates 
staff member through telephone .Interviews; and 

• The Small Projects Study of projects of $s/00Q or less and, having fewer 
than 30 Indian students, Including document reviews, telephone sjrrt^rvlews^ 

" visits to three sites. ; u ; f - , 




As an additional method of filTing in the gaps, in data that came to light 
after the fall visits, the above mentioned Field Staff. Summary forrh was 
developed and given to each team. This form was filled out at. each .site by a 
team member. It provided, the field data' collectors with a format for writing 
down/their observations and insights for each site they visited during the 
spring. This' additional information was useful In helping to form a more 
comprehensive arid* complete picture of the overall Part A Program, and ^S?%lso 
helpful In providing supplementary data to the qualitative Inquiry. 



The fall qualitative data collection focused on topics relating to the Impact '■ 
of Part A which were either not addressed or only weakly addressed by the 
survey instruments in this study, and by quantitative research methods in 
general.- It provided information for determining relevant focal topics for - 
each site for spring qualitative dlta collection, and provided other data 
needed during spring data collection. ' • - « 

Prior to fall data collection, field staff members Were given and became; 
familiar with a set of topics for qualitative data collection in ;the 

; ' ^ , J r— DeTELOPMENT ASSOCIATES, INC* 
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field* These tot 
instruments, rec 
In addition, fi 
which 4 Were ^elc 
addressed bys 

Foil ©wing the 
brief narrative 
tlon given to 
they encounters J 



it 



1 

o Pa 



^ although addressed 1n the survey °[ 
ml .ualltative data to be addressed adequately. 

/ ;ructed to remain alert; to si tf phenomena 
i dvltles and Impacts, but did not appear to be 



lection, field stiff mfemfiers were asked to write 
dir i: (a) the ^et of topics for qualitative data, bollee- 
i to entering the site and (J>) any site phenomena which 
jastdered worthy of further qualitative' investigation '~ - 



during the spring data collection. 



Frcpi these, a finite set of research questions relating to the impact of Part 
A was generated by the Technical Advisory Panel and study staff, The * 
questions wer t e sufficiently narrow mo that they coul d be addressed by brief, 
focused! qualitative Investigations* 



In .the' spring, topical qual itative Inquiries Were conducted for an identified 
subgroup of 31 of the 115 projects in the sample. Data collection' was carried 
out by , the same field staff who adninistered the structured survey .Instru- 

k - -mm 

ments. Spring sites were subgrouped according to ^relevant research 
questions. This was, accomplished by: ^ 

• A' debriefing of field staff following fall data collection with questions 
concerning salient site phenomena; 

• An examination of the proposed impact questions by the flfld staff 1 n ^ 
order to determine which might be relevant to their sites; 

• An examination of the unprocessed data from the fall data collection; 

• Information from 1981* applications; and ■■ ( 

■ . ..s 

• A final revieur and selection of topics by the Technical Advisory Pan£l 
during a scheduled February 1982 meeting, . v \ 

v ' . • ' •;. • • ■ 

In summary, qualitative data relating to the research question(s) selected for 
each site were gathered, analyzed* and written up by the field staff 'at that 
site/ A single study^ staff member acted as idl tor/analyst for eaqh of the 
toplcaT studies from a.1 1 sites, being responsible for reviewing each study, 



\ discussing' weak points with authors, and performing interrtal syntheses of the 
qualitative studies with the results from survey data analysis, 

■ . ■■ ' ' '-ti ■ ■ " ' ■ . * * 

L Data Processing and Management %^ ' 

* « = ■ *. a * * * ** 

By the.grfd of May 1982/ all site vrtits; were completed and the majority of 
data was logged-in. Missing data were determined, "and attempts were made to 
collect It during the summer months. of 1982, < v 

Data processing of the fall data was completed In January ^nd began again 1n 
April as the spring data started to arrive. Each bmpl eted questionnaire was 
logged-i n by site, checked, and coded. Coding manuals were developed foi* each 
data collection period, together with response categories for each op^n-ended 
item 1n the 7 interview guides and questionnaires. • 



^ All interview, telephone, and ddta recording^ and instruments were coded and 
edited by trained personel . AIT Instruments were reviewed at several levels 
for incomplete or unreadable responses and- Inaccurfc te, out-of-range, implausi- 
ble^ or logically Inconsistent entries. AJ1 manual editing and insertion of 

• * ... | . ^\ - 

updated information was done by trained coders urtd&r* supervision." Such 

Infbrmaton Included Identifiers and sampling weights. All coding was 

conducted under formal , ongoing supervision, and periodic review of .the work 

-Sone. - 1 ' . ; ^ , f • \. 

sljj) pen -ended responses were coded after research ana1_ysts versed in Indian H 
education issues reviewed responses and developed coding frames* This was 
done for all relevant open-ended interview items* Coding frames were reviewed 
for validity, uniformity, and usability, and revise d as needed before given to 
coders. Coders were trained In the use of these manuals before beginning 
full -scale work. 

& . ■ * * * 

In particular, there were three instruments which 1 deluded a large number of^* 
open-ended questions: . (1) parent Interview forms, 1 2) the tribal /community 
leader interview forms, and (3) the parent corrmitte^a questionnaire. These 
three forms represented the thoughts, observations , and' feelings of the^adult* 
members of the- Indian or Native community in which ^fche project and school 



district was located. It was deemed important that the interpretation and 
coding of the Individual answers to ^thesa opsn-ended questions be conducted "by 
American Indian/Alaska Native eode**s who wars famil far with the /art A 
Program, and the possible subtleties in some of the questions answered^ from an 
Indian perspective,., tjius, arrangements Wte made with the Center for Indian 
Education at Arizona State Universa lly to have Indian graduate students do the 
coding of these instruments. 

?ive of the ^oders^ were enrolled ih am spic1-aTT seminar for 4 which they were 
given academic credit, A three-day —training and orientation session was, given 
by the study's deputy project director, The .review arid coditig of these forms 
took place during a four-week period in July and^_August, 1J82, In general * 
the processing of *. the completed in«tcruniefits occurred as the data arrived .from 
the field. Data processing conclude^ d by the end of August, when data analysis 
procedures began. 

- k> y 

Following manual edit aVd coding* pro«ecliiris, all forms were grouped by type 
and turned o^ef to a"; keypunching f^c- -illty (Mailing' List Systems, Inc. ) , 
accompanied by^ detailed keypunching - i nsir|et-E' ons for each form. All Instru- 
ments were desi gned^f or direct keyjjUtHiehing,' In view of the diversity of 
survey item formats and number of da' -ta collection InstrumentSi JQOS indepen- 
dent verification by the keypunchlrig facjlty was performed. 

Computer editing was conducted us1n9 -specifics editing instructions devised for 
each type of form. This generally ceonslsttd of a series of checks for com- 
pleteness, accuracy, internal const s— e^ey,, and out-of-range values. Several 
sets of editing checks were made ot% ersacl) ffl<=?, and, after each, analysts and 
research aisistants- reviewed the typ#=es of Issues which needed resolution. 
After^several such computer flTUT checks, eaef-i file was considered fully usable 
for composite Index development, co**r relational analyses, and ether analytic 
purposes. I , 

During the period of April through September 1982, data analysis plans for 
each study, component were drafted, teevlwed, and finalized. Several Indian 
educators,, including members of ttl*\t stub's "n"ectin1eal Advisory Panel serving 
as consultants w the study, assista* 4 In the development of these analytic 
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plans, Indian educators and research specialists flayed proles in the 
- following areas: • > 

• Developing analytic plans; 

• Conducting analyses; ; ' 

• Interpreting analytic, results; 

• Reviewing and commenting on drafts of results ^nd -reP<?Mi; and 

• Writing sections of the final report and study monogr^M* 

«- i 

4. Data Analytic Techniques Used . -. 

As cited earlier, the general purpose of tills evaluation WAJto re ^mch some 
conclusions about the effectiveness of the Part A SntHleni% fit Prog ram. The 
broad study issue was defined as follows* 4 ■ , 

To what degree does the Indian Education M Part i=VEntiti §mt Pr ogratn 
contribute positively to meeting the special educational aMcul tu T*all,y 
related academic needs of American IndiaMliska N&tlye cut Idreo a jid ysuth? 

The analyses focused on three objectives drawn frerai this &Nad stu cty i$su£; 

v/ 

„ § To accurately and sensitively describe the ran^e of PiHb project 
objectives, target groups, cultural activities^, and prAjjrain fu ndirtg 
patterns; 

e To determine the.nature and extent/ of "Part A Program inlets o n Indian 
students r Indian parents, and local school district 5; $3 

• To determine what, if any, changes in legfslairSon sr f%0tiQ ns will 
provide a more systematic and effective approach to mating tft « 
educational needs of American Indian children, 



2 Please note that, in addition to the exposition of tsehniflW Merited in 

this section, each chapter of the Final Report incl u^esi, SpM flc detail s on ho* 
"particular techniques were applied to the evaluation issues. 
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Key analyses used I n this evaluation were, based on a seH^sof anal, ytic 
memoranda whlcli recommended analytic techniques and variables to bte used and 
discussed r*ef cvant issues* The actual analyses used In thepluatelon began 
with traditional approaches to Understanding the descrlptlvi characteristics 
of .variables* AnaT^ysts then proceedfd to use correlational and njof* - © . 
sophisticated anal 3^ tie procedures In an attempt to understand the 
relationships long those variables^ Analyses were conduct^wlth several 
types of r^spooden-£s t to- triangulate findings and thus attempt. to ftoave a 
pattern converge 

Wherever possible, several approaches were used to determine that a - sufficient 
basis existed for concluding that a statistically significant and/o &r meaning- 
ful relationship existed. These included Using the f 0! 1 owing appro^aches 2 the 
presence of statistical significance (alph f a = ,05), percentage differences 
occurring between comparison groups of at least 10S # arid measures 0 explained 

2-2 / ' 

variance, such as or R When using bivariate or multivariate cor-rela- 

ti on coefficients* respectively, and overall appraisals of to much of a 

difference wasa pr^actlcal one which had programmatic significance. 

The latter decisions were based on familiarity with thm dimensions a being 
measured, and the ^ n^tructional , cultural , or other project ictlvit „ies which 
likely would have accounted for the obtained result, Such appraisi^ Is for 
practical significance helped place the findings In content and reds uced the 
chance that meaningful findings would be overlooked, '* , 

To that end, a series of balanced and comprehensive Impact analyses was 
conducted, for the most parts the key unit of an&'ysi& w$^p Par^~t A Indian 
education project, as the entity which was oper a.g^a funded progffcam. 
However, It alsowas recognized that other types of key d^ta sourcess , such as 
students and parents* were Important in their own rl girt' as 1 units of analysis* 
Several Indicators of Impact and multiple data sources vW^uiSd on almost 
every dimension, to : , (a) detect the extent and form of irflpaets occtairrlng on, a 
given dimension) (b ) obtain a consensus f i«om setaral sou rd^s that ifioipact had 
occurred* and (c) clearly understand the role and extent of Part A ©effective- 
ness, * ■ , 
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In general, analytic techniques* were used in this study to establish relation- 
ships between program variables and measures of impact on a\yaMety of 
dimensions or several data sources. Such techniques were necessary in view of 
the Part A Program's fairly, small and diverse mix of services 1 ri some dis- 
tricts* A number of approaches were used to analyze the Information 
gathered. Some approaches w^re qualitative while others were quaptitatl ve* 
In addition, some , approaches, were primarily descriptive, while others were k 
quite sophisticated ami represented the state-of-the-art in jmil tivarlate 
statistical techniques* 

The array of such techniques included the following: . m 

\ ' i ! 

t Factor analyses and psychometric analyses , to establish the tot measures 
of particular dimensions, and to confirm interrelationship? across a set 
of predictor variables; 

• Stepwise and hierarchical multiple regression, to ascertain tdi t relative 
importance and best subset of variables which were related to measures of 

^tutoring or attendance levels, relying on conceptual models of what 
\^ predictors should first enter the prediction process; 

• Path analysis causal modeling approaches, which permitted tne assessment 
,of the relative direct and indirect contributions of prdj eGt activities 
and similar variables to various Important measures;' 

§ General 'linear modeling multivariate approaches (similar to multivariate 
analysis of variance techniques), which were used, to slmul taidous.ly assess 
relationships between a series of project and contextual factors and the 
two dependent measures of reading and math standardized test scores, while 
controlling for project parti ci pan t-nonparti ci pant differences; and 

e Meta-analytic techniques which used Indian and non-Indian .test score 

• comparisons q converted to "effect sizes' 1 in z-score format)' so that- trend 
analyses could be synthesized across a series of historical time . periods* 

Htfwever, all of the analytic techniques used In the study h*d certain common 
features, which were the following: /' - _ ) 

© nW technique was deemed relevant to address.a particular major or related 
study question; * ■ 

% Some form of comparison group was used to place the obtained result In 
context; this might have been: other groups with systematically different 
characteristics, students receiving services In a different my or from a 
different source, students or programs, tracked over a period of tfrne so 
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that a trend across time was observable, or^ojher studies^ 1 findings from 
- the literature which provided a basis for interpreting and/or contrasting 
the present study on relevant dimensions; 

# . * * *■ 

• All techniques which were used al so took Into account: the type of imparct 
being assessed, the sample sizes being compared,- the number of comparison 
groups, and the extent and type of.missing data. v 

Also, as is true in many evaluations^ certain other techniques were .performed 
which did not^provlde anymore Insights than''pTreaty/]^tftt<3 fty mors straight 
forward approaches, or which we're not .useful due to vaHout ra^asurertt and 
design 1 imitations. For exampl e , the study employed; canonical T correlation - 
techniques to see if a relationship existed between a set of cultural program 
characteristics and a set of student achievement measures, arr d stepwise - 
discriminant function techniques to better determine if level s of patterns of 
student attitudinal scores were more characteristic .o'f progrfejn participants ^ 
than non-participants, Although sometimes useful , In this sfcxidiy these , 
techniques did not improve upon the utility of Impact evident e obtained 'With 
less complex approaches * * - 

The latest versions of the state-of-the-art computer data pro messing software 
packages. Statistical Package for the Social Sciences, (SPSS^i'el ease 9. 1 ) and 
the Statistical Analysis System ( SAS , release 79,6) were used" for conducting , 
virtually all descriptive and analytic techniques, with the exception of tne_ 
Hewlett-Packard graphics system which wqs used for preparing certain line ' 
graphs, ^trend 1 i nes , and bar graphs | particularly for attendance analyses. 
SPSS and SAS provided the flexibility to accomplish a series of stucjy tasks 
Including data base editing, file merges, and statistical analyses such as 
cross-tabulation^ subgroup means, multiple, 1 inearr Wgresslon, general linear 
modeling, and even more elaborate techniques. 

Most analyses vbre performed on a data base that was designed to permit both 
file (i.e, , particular respondent) and subfile (i .e- , subgroups of respond- 
ents ) h analyses. This is because certain analyses were better prepared §t the 
project level , and other analyses at the student, parent, or other individual 
respondent level* The number of students Included in various analyses of 
attitude, achievement, .and attendance relationships is Included in Table 4-2, 
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, .. . j • • — 

NUMBERS 1 OP STUDENT^S' SUPPLYING E I NFQRMATIQN BY TYPE AND 'GRADE' RANGE" 





Number of 


Students 




Type of r jnfok4nat1dii:i. ' , — / * 


4-6 


7-12 . . 


Totals 


A. Attitudinal Data 








Studtnt QueftlQWtres - .f&IIV 
, ■ Student Ouestlbrifiifres - spring 
', Both „fal 1 and spring 


5,328, 
5,173 
'4,255 


1 8,149. , . 
7,366 
5,665 


13,477 
12,539 
9,920 


B s Attitudinal and Actiieytem^nt or Attere dance 


Information 






Fall, spring attitudinize 1 arid achievement 
test scores 

Fall, spring attitydlfitol ahd 
attendance fiat a ■ — , 

Fall s spring attitydina_l % ftcfilivtmera % 
and attendance data — 

** 


2,688 ' 

3,280 

1,622 


2,585 . 
3,317 s 
1,538 


5,27-3 
' 6,597 
. 3,160 


C* Tutoring information 


K-6 


7-12 




f Fall- only * * — 
Spring only - - • 
Both fall and spring ./ 


189 " 
129 
1 ,693 


, 211 
119 v • 
,1,187 


400 % 
248 - . 
' 2,880 .- 



' (a) The Role of Dgsgrlj tjv e Tishnlqu es . These types of analyses were 

performed dn"»ll* fnh* pact artis. The techniques were useful in their own 
right for dctwinl ng tNeharac ten sties of Indian/Native students, 
program operations "^and other fac -tors. However, they also were invaluable 
for gaining a full understanding" of how widely the data varied, so that 
other, more Indeptsh * techniques could be applfed,; / 

The descriptive t^c hniquSsused in thev study included mos^Qf/the widely 
used approach, ih eluding; frequency and percentage distributions, 
means, standard dev iatlorts, coun x$, medians, and ratios, graphic 
techniques such a$ _iar graphs* t rend lines, and scattergi*ams were used as 
relevant to hjahTfg ht the presentee of outliers and dispersions and 



particular trends across time or across, comparison groups , s^o that their 
r Native levels could be determined. Cross-tabulations, correlation XI 
coefficients, and factor analyses were' used to establish if *and how 
certain variables were related to each other. Other, techniques were less 
quantitative*, and represented tabulations of predominant topics mentioned 
.in narrative responses >^y respondents^, so that prevalent 'themes coulld be 
Iditftifled,* and the range of discrepancy (i.e.* how many respondents did 
not share that central opinion) coul d be determined. 

• * ; v - v ,; : v-.^-. t . " M • \ m : - ' ■ ■ ' 

Some of the areas of investigation which ^^marily were examined using 
descriptive techniques, due to the topics with which thty dtalt and the 
levels of measurement suitable for dealing with such topics, were the 
following: * v- 

v • Project characteristics and overview summaries of local project 
operations and emphases; . 

• Needs and objectives of local projects; assessments of the 
appropriateness of project emphases and activities; x, 

• Descriptions of cultural programming and activities; appropriateness 
of cultural activities* , . 



• Tutorial and, related activities directed toward enhancing the basic 
academic skills of Indian students; tutor, student, and program 
characteristics; . ■ U:-:\ ■ '.:■>■'■ 

• Parental involvement and relations with the Indian community; 

• Perceptions of parents, community members , and tribal leaders of 
project operations and effectiveness; 

• Project representatives 1 perceptions of the Office of Indian 
Education, the types of changes in Part A rules, regulations, and 
legislation which appeared useful to them, aritf preferences for various 
approaches to determining student eligibility for services; and 

• Levels and longitudinal trends in employment of Indians and Alaska > 
Natives in various public school positions (administrators, 
principals, teachers, teacher aides, and other personnel), 

A series of cross-break variables was used to detect between-group 
differences in these topic areas (see Table 4-3). Project and student 
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TABLE 4-3 

CROSS- •-BREAK VARIABLES USED IN DESCRIPTIVE' AND IMPACT ANALYSES 

; , ■ L, CZ L- 

Av Project Settling 

1. ^Percenta^igs Qf^Ind+an students (density) in district (less ,than Si* 
• * 5,1-701; ever 70S) 

2, Residence of most Indians 1n district (on or near reservation, othe 
rural, umirban/non-metropbl i tan, metropolitan area) , 

3* More thiKin half of project participants are of the same tribe (yes/n 
' ^tri bal ho omogenei ty measure 

4. Seocultywiral regions (12 categories) 

5, TAC regioons (5 categories) 

Br Project Effbi a-ses and Activities fc ■ * ^ 

1, Number or*f students in district " ' - a , 

2. Number or f Indian students in district 

3. Number dt f participants In local program 

4, Part Afuunding level 

C, Project Eoiphsages and Activities - 

1., Project s services '(whether or not stress on: culture, counseling, 
home-schocol , or tutoring) 

2; Percentao ge of dollars coming from Part vs. other sources 

3, Project ogbal s (whether or not stress on: ^academic basic skills, dropout 
and absinnieeism, counseling and attitudes toward school , cultural , 
students* 1 self -concept, drug and alcohol abuse education, 
medical/o dental /financial support, Increased parent/community Involvement, 
or 1n-$errrvice training), 

D. Studen t-Uvin Va r j abl es 

1, Length p*f time In program _ 

2, Amount oft-f help (extent received) from program 

3, Type ofhlhelp received (program component) 

4, Grade lawval (or grade range of: 4-6, 7-9, 10-12); 4-6 and 7*12 also used 

5, Age* f 
6* Sex 

7, Home lanoguage". use 

8, Pride in Being an Indian 

9, Percelveod importance of Indian topics 

10* Socioeconomic status (whether or not on free or subsidized lunch) 



level data were routinely analyzed ifi tenris of these variables, although 
results of these analyses were provided only where meaningful * , 

(b) The Role of Impact Analytic Techniques, - In essence, there were three ; ; 
major domains of Impact. There were impacts- occurring to: students, 
parents*, and gommuni ties and schools districts, for the most part, the 
descriptive and analytic techniques used to Investigate each domain were 
more similar to each other than across domains* Because of this; these -/ 
three domains of impact were used below as the basis* for discussrng What 
analytic techniques were applied throughout the stucfy. 

(1 ) Student Impacts. and Outcomes. The descriptive an<^ analytic techniques 
used with this domain generally focusei^on determining what, benefits 
students had obtained from participating in program activities. 

" - - r i 

» " * * * w : , -, 

Historical Trends 1n Academic Achievement Among Indian Students in 
Public Schools ; 

These trends were assessed by using meta-analysis techniques to 
synthesize findings from a literature review of past studies of Indian 
student performance prior to the enactment in 1972 of the Indian 
Education Act* compared with subsequent levels of achievement. 
Studies conducted in four decades (1950-59, 1960-69, 1970-79, and 19fiB 
to the present) generally Involving public school settings were 
obtained. > 

Five strategies were used to locate studies, references, and test 
data* An ERIC search was conducted which located 90 relevant articles 
for the late 1960s and 1970s time periods. The Native American v 
Research Information System (NARIS) maintained by the University of 
Oklahoma produced over 800 abstracts which were reviewed and which 
provided coverage for the 1970s time period. 



3&te Appendix 4 (Methodological and Conceptual References) for citations 
dealing with the techniques used In this study-^j^ 



Indian education authorities suggested a number of key reports and 
past studies which become the primary Information sources for the 
mid-1960s and earlier periods. A series of contacts with state^and 
local education ag&ncies yielded 20 reports or data sources, only two 
of which had sufficient sample sizes and relevant analytic findings. 
This source provided late 1970s and early 1980s data. Finally, the 
present stvtdy of 6*425 students in 77 districts provided cross-sec- 
tionaj reading and mathematics ^standardized achievement test score 
data. •■• . ' . ' * 



From this extensive set of sources came 100 promising studies, of 
which only 1 6 were suitabl e. ^Da ta from these were used for the 
meta-analysis by applying t He-foil owing techniques* . . 

• Percentile scores were concerted to z-score equivalents or effect 
sizes, using normal curve>;conversi on tables; 

« • Standard deviations were flsed from the, Indian student group, if 
the comparison group stanjjard deviation was unavailable; and 

• Test norms were used when a study did not u^e a local comparison 
group, th6 preferable analytic unit for meta-analysis purposes. 

Thus, standardized achievement test score results were then compared 
over time periods using effect sizes (In z-score format) reflecting 
the relative magnitude .of Indian student academic performance compared 
to baseline groups (such as Indians vs. non-Indians, or Indians vs. 
all students), The "meta-analysis approach" pioneered by Glass and 
his colleagues (e.g.. Smith and Glass, 1977), 4 and now widely used 
fqr synthesizing research evidence from diverse studies by using a 
common metric for comparison purposes, was also used here- Effect 
size comparisons across time periods were performed and graphed for . 
the purpose of clarity. 



^Sml th, M,L. & Glass, G. V. Meta-analysis of psychotherapy outcome studies. 
American Psychologist, 1977, 4 32, 752-760. (Also, see Edu cational Research 
Journal, 1980, T7l 419-433.)-" _ — m : — " 
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Participation in Tutoring Programs 

Academic Impacts assocl ated wi th u parti ctpatl on Ih tUfeHng programs 
were measured by obtaining tutor ratings at the oh^t^and end of the 
school year of student participation on their v levels of reading, > 
mathematics, writing, and social studies* Tutors were ^al so asked to 
rate the students they were tutoring in thei r 1 evel s of sel f -confi- 
dence , interest fn school wort,/ classroom attendance, and conduct in 
school * The major impact variables, ratings on reading and mathema- 
tics, wipe analyzed using a very broad range of project* tutors and 
student ^rJ abl es . Two-way cross-tabul ati ohs and mul tiple cross^break 
analyses were f 1 rst performed*to detect i nteractfbiis and relatl on- :- : 7't- 
ships, followed by a series of stepwise multiple linear regressions. 
Separate regression analyses were theri conducted using po^t-test 
mathematics and leading scores as dependent variables* In all * 
regressions* the corresponding academic pre-test (onset of school 
year) measure was forced into \the regression equation first, as a 
eovariate which controlled forHnitial levels of a§&demi£ performance. 

Ratings of Project-Related Academic Gains by Teachers, Staff t and 
Parents • 4 

This Information was obtained by asking teachers of Indian students, 

. .. . 'f . • ■ 

project staff, and parents of Indian students to rate how much the 
project had helped to Improve student performance In reading, language 
arts* and Mtliematlcs, and how much the project had helped to improve 



student grades* Four-point Li kef* t; rating scales were used, The 
scores on reading and language arts measures were combined to form an 
overall language arts scoi^e. Thf resulting Impact measures were %|f^ 
analyzed first descriptively and then morr intensively* : Thus, 
projects on or near reservations were compared to those In other rural 
areas * In urban areas or In metropoli tan areas. The amount of effort 
that each project expended on basic skills instruction In language 
arts and mathematics was alio used as a key cross-break variable. 
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Thpse formally spending effort tc* improve baslq skills instruction 
were therefore compared wi th thos§ projects v. reporting no hours of 
; 6ffdrt expended on basic skills, projects ■ Weg& - al so, class1f1ed%y 
ratings" of impact on student grades, and ^fee projects ,in each - 
category were compared in terms p/ ..the amount -of effort spent %n basic 
skil l s and other academic instruction. General l inea^ model in 
correti$1 oiial procedures were used to determine what 'factors were most 
associated with the mean impact rati ngs ^obtained from various 
cl assi f 1 cation vafiabl es. These means were al so presented in mul tl pi e 
bar graph format so that, relationships, across the project c^l^sifl ca- 
tions were more clearly evident. v ; ^ 

Ratings by Students of Project-Related Academic Gains ^ ^ ■? 

Indian/Native students in the grades 7*12 study sample were asked to 
rate how much they had learned from the project teacher or. tutor who 
Staught two subjects in which they had recei vet^ instruction during, the 
J981-82 school j^ear: mathematics and reading or English language % 
:Wts. Ratings were based on S-point Likert rating, scales. Data 
sources for cross-breaks also included relevant information from 
teacher Ind staff interview instru^ntts, and project descriptor f opts. 

These! students 1 impact ratings were analyzed by school . district 
location and the number of hours of project effort on reading arid 
in^thematics. They were also compared with the Impact ratings of 
student gains In reading and mathematics made by teachers and project * 
staff in the same school district, Thus the Impact of Part A on 
academic Improvement was analyzed from the distinct perspectives of 
students, teachers, and staff, all associated with the same project* 
For additional analyses, students were grouped by the levels of their 
ratings* The relative number of hours of effort spent to improve 
reading and mathematics, compared with performance In other academic 
areas, was determined, as was the relative extent to which students 
attributed learning to- Part A teachers or tutors, compared with 
regular classroom teachers or reading and mathematics specialists. 
The unit of analysis was the secondary school Indian/Native student* 
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0 General 11 nean modeli ng procedures were used to testjjM fferences . • ; sf 

' between groups and whether they were assodated-wlth certain X 

. •. \ ■ • • : •. • . * ,• .- 

hypothesized variables. , J ** • 

Reading and Mathematics Test Score Patterns of Students 



A crdss-seetlbnal design was used with standardized reading and 
mathematics test scores, collected from district files having a ' - 
testing program in force during the Sprang 1981. /These test scores / 
were converted r te T-seore format (based on publishers' standardization 
group means and standard deviations), and then used as dependent 
measures. ir w : ; .•. ; ..- r .,.....-..,.,._..._ J .>.....,^:, • .-.,>,^^. 

• • - ' ; -.]-.-'. " - •■ " " 

Descriptive analyses dealt with understanding the levels at which 
Indian/Native students were currently performing in' reading and 
mathematics, and to what extent* they differed from the general student 
population. Various means, frequency distributions/ and cross-tabula- 
tiony were used for these purposes. Impact analyses used multivariate 
general linear modeling approaches to test if a series of 
student-level and project-level demographic/background^ contextual,- 
attitudinal and project operations variables were mean! hgfully related 
to reading and mathematics scores. Since the achievement. measures 
were correlated, multivariate modeling techniques were used to test 
whether these predictors were significantly related. to both dependent 
measures on an overall basis., 

Analyses wire conducted, using the student and then aggregated scores 
to the project level as the unit of analysis. The general linear 
modeling statistical" tests were further strengthened by Including _ * 
ihree covariates which controlled for parti CI pant- 
nonpartlcipant differences in- geographic region, language used by 
students at home, and grade level. Analyses were conducted at the 
elementary (grades 4-6) and middle/high school (grades 7-12) levels, 
to determine if findings held for both grade levels. The amount of 
explained variation was examined to determine if meaningful rela- 
tionships were found; the direction of findings was also examined. 
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Longitudinal Attendance" Patterns ^ „ ^ ' k' 

TPfnds in attendance of indi an/Native students oyer a = four-year period '* 
. were intensively Examined, The mean attendance of H:he Indian students ' V* 
was compared with known and^ estimated, national attendance' dstta. , Mean : 
attendance" levels of- various subgroups were analyzed for trends%aeross _ - 

^ the four years and compared with . the overkll Indian- student mean days 
of attendance trend line* Indian student attendance trends were 
analyzed by a slries of variables, including: region, project 
population density and project location, school ^district size, grade 
level, school district size, sex, socjoeconomic status, hours of 
^ project effort per student pen year ex^ 
and proportion of effort expended by the project during the School day* 

- r to improve attendance. Attendance levels were also correlated with ; 
standardized reading and mathematics achievement test scores^ parents 1 
general satisfaction with their local projects, and parents 1 . 

v perception of school personnel sensitivity toward Indians. \'x .\ ; 

Trend lines were used to analyze attendance patterns Of meari * 
attendance levels for particular subgroups of Interest, hierarchical 
model multiple regression and linear model procedures were used to ,/ 
determine how strongly selected variables were related to 1980 ■ , 

attendance of Indian students, after control l ing for other factorsl^A 
regression analysis also was used to predict the national mean days of 
attendance for students in the United States beyond the year 1976, 
since national -level data were not collected for the 1977-80 time 
period. -Jhe trend line for Indian students was then compared with the 
national trend fine. All analyses were based on the student as the 
unit of anal ysisVsComputer-genera ted graphs were used extensively In 
this phase of the study* ta 

Project Impacts on Student .Attendance 

Ratings of the extent to which the Part A project had helped to 
Improve attendance of Indian students in the school district were 
col 1 ecte^Trom teachers of Indian, students, Part A project staff, and 
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parents of Indian students, These ratings were combined within types 
of respondents to obtain average, project-levels; of Impact on attend- a 
ance as reported by the three respondent 9 groups, Thfey were then * 
analyzed by school district geographic location, and average hours of 
project effort per student pfer year expended on Improving attendance, * 
General linear model, and correlational proci^ures were used to " 
determine relationships of various predictors jlhd khese project-! eVel. 
ratings of project effectiveness In Improving attendance?^ 

Post-Secondary Knowledge, Aspirations, and Experiences 

Students in grades 10-12 were asked a series of questions " to ^sess. 
what they knew of post-secondary educational opportunities, and what 
their aspirations for, such education were. Knowledgeable sources In 
local projects and the conmuhity also were contacted to supply 
information on what former high /school • students (thort who, had 
attended school during the last ten years) actually were doing with 
their lives at present, A series oft descriptive statistics was used 
to portray the typical patterns ^€f key variables, by themselves and in 
relation to other variables. Impact ara|yses sought to relate the ^ 
type and amount of knowledge, aspirations^ and actual experiences to 
project contextual and program opera tiori factors. Trend lines were 
examined to determine if Indian students 1 ^gh school graduation rates 
were related to the operation of Part A programs, ■ v 

■ V'-;, Y ' ' ; '' • * •: : ' ' •; v 

Self-Concept and Attitude Toward School % • 

During the fall and spring data collection periods, students In grades 
4-12 were asked a series of rating scale Items organized into attitude 
scales tapping such dimensions as academic self -concept, attitude 
toward school, global self-esteem, and pride in being Indian, The 
same Items and scales were adnlriistered twice to determine the levels 
of attitudes, If students 1 mean attitude scale ratings shifted over 
the school year, and whether or not these levels were associated with 
participation in project activities and various personal characteris- 
tics, such as socioeconomic status or^g^e level. 

t * * ■■ 1 ■ * • t . 
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Descriptive statistfeff focused on understanding the current l evels -of 
. * Indian/Natl ve sel f -concept. Fall and springs scores Were descriptively 
* analyzed for their dispersion, typical value, and 1 ntercorrel atron 
with projects-level aird student-level variables.- Both scale scores and 
Individual items were, examined, ■ j . 1 

n . , * ' : • - ■ 

■ <(2) Project Impacts , "These -analyses pHmarlly dealt with suitability #nd 

•' ■ - * - - \ # • 

* * effectiveness of project operations, * •■ 1 -A • 

. \. -V ■■■ v:: .: : : . . v : •• ' 

^ Appropriateness of Local Needs and Project Objectives 

- • ? These analyses focused on assessing the appropriateness of the 

activities being performed by local projects, 1n terras of whether or 
^ not the special, educational or culturally related academic needs of 
Indians/Natives^ were being effectively addressed A number of types 
of Information were gathered from project directors regarding their 
needs assessment procedures and what types of needs are eonsi dered the 
most salient and deserving attenWpnv These variables were analyzed , 
by using descriptive statistics such as frequency distributions, . ^ 
means , percentages , and selected cross-tabulations using key 
cross-break variables. Other sets of analyses, related project needs 
to objectives and activities on a project by project basis ^to estimate 
the degree of appropriateness of project operations. Categories of 
expressed needs were separately analyzed to determine If the 
objectives and subsequent activities of projects for certalti needs 
were relatively more "on target" than were their emphases for other 
needs. 

Attitudes of the School and Community Toward Indians 

Questionnaires administered to community and tribal leaders, pareht 
committee members, project staff, and project directors contained 
Items dealing with current levels of sensitivity of eight different 
groups (e.g., principals, elementary school teachers, school boards) 
toward Indians, and, whether the respondents perceived any shift in 
such attitudes in the last few years. These rating scale 



Items dealing with school climate were analyzed to determine what the 

current levels of sensitivity were overall*, and for particular types r 

of individuals,. Analyses were* also performed to determine If a. - 

consensus existed '(across types of respondents) that a change in 

climate had ^occurred^ These ratings were therefore analyzed by . 

respondents at the dlstritt level, and by selected key variables. The 

* • latter approach was used to assess if climate shifts had, occurred 

3pW " relatively more often 1ft certain types of projects than In fathers and 

l > % thus might be associated with , project operations* 

. ^ .. " . ■ ' 

. _ ■ 

^ (3 > Indian Parent and Community Impacts * Deserl ptl ve stati sti os were , 

/ extensively used to highlight what project act! vi ties were geared to 

enhancing parent and community Involvement Jn local program 

^ operations, and determine the nature of that involvement. 



More elaborate analysis techniques were used to determine what factors 
affected parents' generil satisfaction with the Indian education 
project, Thus causal modeling path analysis techniques were used to 
test a hypothesized*! inkage of six factors, These factors dealt 
'with* communication from the project* to parents, parental participa- 
tion In project activities, paren€ committee Involvement in project 
operations, sensitivity of school personnel toward Indians, parental 
perceptions that Indian students' cultural needs were being met by 
project activities, and parental perceptions that the project helped 
to Improve Indian students' academic performance. 

Each causal factor was assumed to have a direct effect on the level of 
parents' general satisfaction with the project, while each predictor 
was hypothesized to be affected by all other predictors preceding Its y 
entry Into* the regression equation, Thus path analysis findings 
Indicated the relative strength of each predictor (its direct effects) 
and what Indirect effects were contributed by the other predictors due 
to their own correlations with causal factors, The sum of these 
indirect and direct effects equaled the total effect of each factor. 
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(c) Programs Currently Serving Indian Students • '% 

Another focus of the 'overall study methodology was the Alternate-Resources 
Study; this study was designed to estimate how many Indian students were 
being served by various programs (state, local, and federal), with what 

' .resources, levels-, and activities were being provided. Data from four • 
" independent samples of Part ^eligible 'schools/districts -provided estimates 

• of this Information, . * ' . 1* , * v 

Analyses dealt with seven types of, federal, state, and local programs to 
clarify the differences across such programs^ and pinpoint which programs 
were used relatively more than others to serve Indians. Thus, frequency 
distributions, percentages, means arfa medians; cross-tabulations, arid 
related statistical approaches were used to provide summary statistics on 
these topics. , 1 

5. Limitations! of the Methodology & *~ 

Finally, this report should note that there were a number of factors relating 
to the Part A Program and tins evaluation which limited the ability to draw 
unambiguous conclusions from the study's results. Among the more important 
are: . • 

• The Part A Program is merely one of many special programs which serve 
Indian students. Chapter 1, Title VII, Johnson D'Halley, and other 
federal programs all provide special services to Indians, and 1t Is 
virtually Impossible to separate the effects of these various programs 

• from each other and from trie regular school program. 

• Part A projects are relatively small, and they provide a diverse mix of 
V program services. Thus, Impacts may be quite particularistic and may be 

hidden by grouped data or the application of general criteria. , 

• The evaluation was conducted many years after most Part A projects were 
initiated. Many of the greatest impacts of Part A projects may have 
occurred prior to the evaluation, and projects may simply be sustaining 
earlier gains. The lack of pre-Part A baseline data on many variables 

, made interpretation of such effects problematic. 

• The evaluation was limited in its time frame for collecting data. Fall 
and spring measures were collected on a number of student measures, but 
the average time between measurements was only approximately six months. 

( ' - ■ . " ■ / 
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This may not have been sufficient time for Impacts to occur on some 
student measures (e.g*,* attitudes and tutoring)^ r r ' 

• It was dtffJcul t^tcT*&lass1 fy students based on thel r 1 evil s of 
participation In Part A activities. Some students were not aware of 
which supplemental activities were related to Part A* and project staff 
at times had difficulties Irt specifying which, students had taken part In 
open programs such as cultural assemblies* In addition, some project 

x activities Were provided to the school or community as a whole, so Jthat 
the subject groups for such were difficult to specify, The effectiveness 
of staff and personnel in providing these services, or the responsiveness 
of students to these services* also could not be directly assessed. 

' ■ . ; ^ * • . ■■■■ - • . . " = ■ • 

• Some of the Impact measures had not been previously ^normed on Indian 
populations. The .evaluation was thlis partially exploratory in nature* 

; providing new evidence on response characteristics of Indian students, * 

Some of these limitations are particular to the evaluation, but many are 
generic to educational evaluations, overlap of programs* diversity of 
objectives* and lack of baseline information' are Issues that confront many V" 
educational ©valuators, Respite these limitations; however* the evaluation 4s. 
believed to have provided reasonably accurate descriptions and Insights Into 
the Part A Indian Education Program and its outcomes. 
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■ ;-• '; -APPENDIX 1 

MAJOR AND RELATED STUDY QUESTIONS FOR THE IMP I AM ». 
EDUCATIOM IMPACT EVALUATIOfl OF PAR*f~ A PROJECTS 

- - - , . ■ . T" "" ■./ 

Broad Study Issue 

To what degree does the Indian Education Act ..Art A Entitlement Program 
contribute positively to meeting the special educational and culturally related,,; 
academic needs of American Indian/Alaska Native children and ^&jth? 

Major Study Questions 1^ _ ^ ^ ^ .% j-^>U ; -;...,.u^^ 

V, What are the organizational, fiscal, and huran resources avail able to Part A 
projects? How do projects utilize these resources? 

a. What are the size and characteristics of projeer-tr staffs? To, what degree do 
projects employ staff who are American Indians or Alaska Natives? 

: . - & " ' i 

- : -- • " . . S . ■ . . : . . . . = . ' ' • • ' 

b. What resources are utilized by Indiafi Edycat^ora. projects funded under We 
Part A Entitlement Program? 

e. To what degree do projects use funds other thar* those granted them under 
the Part A Program? 

v ..... . . . ' ' : ; V 

d. How do the actual resources utilized for Title IV,*Part A Projects compare 
" to the respective grantee budgets? 

e. What types of materials do Part A projects use? 1 To what degree 

do these materials reflect American Indian/Alaska Native' cultures? 

f. What types of assistance do Part A projects receive from the federal Office 
of Indian Education and the Indian Education Technical ^Resource Centers? 

„ In what ways do state education agencies help Part A projects carry out. 
their responsibilities? 1 

- ■ 124 ' . 




ERIC 



g* To mint degree do Part A j> *rojects uj^\ize the human and " otfcr resource** 
. liable In the American Indian/Alaska Native conspunf tl ek hy servt^f — 

h# Wh^ t types of assistance d fo LEAs provide their Part A pH^Nrts? " 

-NoV - can the quality of LEA^ assistance be characterized? - ~ 

■ •■' ■ . = . ■ * - ■ ■ ■ * ' = . • ■ ■. . ' = " 

To fth |1 extent do the objeeti^ ves of project funded under th§ Part 
Erttft j^Wnt Program address tf-fie special educational and/or cMltppaTly related 
yadej tteineeds of Anerl can Irtmmd Ian/Alaska Native children? 

a* WhVtare the objectives of ' projects funded under the Indian Education Act 
Pa^ii Entitlement Progran^? Hew do project objectives rVUte to projefect 
£hirictar1 sties? 

b. To what extent are Part A j project objectives consistent v/1 il the 

characteristics and needs • of the American Indian/Alaska MHIve studen**s thi 
pro Jets serve? " 

c* Ho^ are student needs def i : ned? How comprehensive are needs assessmenltt - 
procedures? ' - 

d. To what extent is there agx reement among project staff, pafent committ^^e 
metiers, the broader Ameri«:an Indian/Alaska Native community, and 
£p£ wiring LEAs about the ^objectives of Part A projects? 

e* Arfi there appropriate stud«ent needs yhlch Part A projects are unable t^rc 
addtfSS? How could the ga^p between needs and services be filled? 

f* To yat extent are project objectives and activities consistent with 
Coflpssienal intent?; 

How_h* WtPart A project activri ties been Implemented? 

a. j/h££ are the activities of Part A projects? / 

=1 * _ — . ■ • 

b. VhC is being served by ParK A projects? 125 : 



c. How are Anerlhin Irwlf an/Alaska KMative students identified? What has been 
' '• the impact of regulations such a«s those relating to the use of OE Form Mo. 

506? ' .. V. .; * % ■" '. ■ " ■ 

d. To what extent are students who are neither American Indians nor Alaska <- 
Natives receiving services funded by Part A Entitlement monies? To what 
extent 1s the provision of servi Tees to non-targeted students Inappropriate? 

e- To what degree are project servi tees integrated with other school progr^tas 
and activities? 

f . To what extent do project activf T ties Include the parents of African 
Indian/Alaska Native students m sd members of their communities? Do certain 
types of activities or project c characteristics lead to higher levels of 
parent and coriuni ty involvements? 

g. What roles de parent coirinitteeg pi sy in project operations? Have these 
changed since the Education Adwe sndment of 1978? ■ 

h. To what degree do LEAs foster e supportive environment for project 
activities? 

iv How do activities funded by the Part A Entitlement Program differ from 

those provided with monies avrfU able through Johnson 0/ Mai ley and other 

programs? Do LEAs exclude AmeH can Indfan/AI aska Native children from 
programs sued as Title I because e of the availability of parallel services 
under the Part A program? _ . 



j. Hew actively do American Irfdian/^ Alaska Native students participate in 
project activities? _. ^ 

k. What Is the quality of the relst =ion£hip between parent committees and LEAs? 

1 . What types of support services ec nhance project implementation? 

How does the need for support ser-r vices differ with the characteristics of a 
Part A project? ' 126 
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m. What has been the quail t£y of project leVel evaluations? To what extent 
*' andl in whtt ways have prwc7^ct5 feuritf thesi aval uatlons useful? * ^ 

What are the impacts of Py;~t A projects on American Indian/Alaska Matfve 
students? .'-.'•;.=.,* • 

a. How dc projects, affect s^ch areas a& student: understanding and . -~ . 
*' appreciation of American triilin/Nati ve Alaska cultures; attitudes towar«cf 

: . school and education; absenteeism; tfr^opout and graduation rates; knowledge 
and pursuit o-f postsecofl^dary opportunities; classroom behaviors; * 
participation in school -activities; grade promotion; and student 
achievement In reading a*n<1 iathemat1 <:s? - - - - — - -- 

b. Do project Impacts on staudMs relate to project characteristics includ— f ng 
size, project resources, project activities* level and intensity #>f stucrfent 
participation* levels of parent or American Indian/Alaska Native cofmnunT-f ty 
involvement, student or ecoiiunity ch aracterl sties , or other variables? - 

c. Are project impacts a fu Miction .of di f faring combinations of activities 
and/or objectives? \ - 

What Impacts do" Part A .pro J Eje cts have on the parents of American Indian/Alaska 
Native children and on the_*»flffrerlean Ind nan/Alaska Native communities projects 

serve?* - ~~ ' 

.. - ■ ■ \ 

a. Hew do projects affect se/ch areas as parent tfnd community attitudes toward 
education and public schc^ol district -s ; parent Involvement In and supports of 
a student's school activ-r? ties, and p x-oject , class, school, and district 
policies and practices? •* 

b. Do varying degrees of par^ertand cowrunity involvement relate to projects 
impacts? ; "' ' ; 

c. How have Part A projects affected the policies and activities of American 
Indian/Alaska Native tHto^sand orcfayii zati ons /Special ly thosjs in that art 

' education-related? 127 • " 
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6, What Impacts do Part A projects have upon their LEAs? 

a. How have Part A projects affected teachers and ether LEA personnel In 

' relation to knowledge of American Indian/Alaska 'Native history and euTture ' 
and these Individuals 1 " senstlvlty to t|e s junLl due characteristics and special 
. needs of American Indian/Alaska Natlve^J^Udents? . / 

b. To what degree have Part A projects led to the Increased employment of 
American Indian/Alaska Native teachers, counselors, and school ■ • 
administrators In LEAs and changes In e^loyrteWt policies and practices? 

C; HoW have Part A projects affected how LEA curricula, especially 
In the areas of U.S. history and social studies, deal with American 
Indians/Alaska Natives?. . ' .' 

d. In what ways have Part A projects contributed to the Increased Involvement 

' of American Indians/Alaska Natives In school or district-wide pol-rcy-makliif? 

e. To what degree have Part A projects affected changes In school or:d1str1ct 
policies or practices to encourage increased American Indian/Alaska Native 
participation in school , extracurricular, and pa rent /community activities? 

7. How do federal and state level activities, especially those of the Office of 
Indian Education, /affect. Part A projects? , 

a. what has been the congressional Intent for and level of Interest and 
Involvement in the Indian Education Act Part A Entitlement Program? 

b. What have been the major objectives, resources, activities and constraints 
of OIE? \ ■ 

c. What kinds of communication links exist between Part A projects and OIE? 
How are these used? 

d. In what ways have OIE activities affected Part A projects? Ho,w 
could OIE play a more positive role? \ 

\ 
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• ..• .. ."'"..i,- • ' ' • . '.V..VV : " * '". V" ;. ' 

e. To what Agree do Part A projects operate under erroneous Interpretations 
of the Part A programs rules and regulations? What actions has OIE taken 

• to correct project misconceptions? ' 

f. what charhges In: 'l.^s1it1^'Mnd v ragHlstl<Mfi would jenhanee the ability of 
4 -the Part A Program to meet the special educational and culturally related 

academic needs of American Indian/Alaska Native students? 

g. In what ways could SEAs positively assist Part A projects? How have Pari;* 
projects affected their SEAs? ' . *' 

, h^ What Impacts have Part A projects rhad on 
.. Indian Education? , . •••'•>,; 

8. what Is The Total Amount of Federal Education Funds Expended by Local School 
Districts on American Indian/Alaska Native students In Grades K-l 2 and How 

Many of These Students Are Receiving 'Various Types of Special Services? ^ * , 

» - ~ -~ ■ * . ■ . & 

a. What Is the total amount of, federal education funds expended by local 
school districts on American Indian/Alaska Native students? 

b. How many American Indian/Alaska Native students are served" by these funds? 

c. What types of special services provided by these funds do American 
Indian/Alaska Native students receive? - 
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r * ; APPENDIX 3 
SCHOOL DISTRICTS INCLUDED IN THE LOCAL PROJECTS EVALUATION SAMPLE 



STATE 

Alabama 

Alaska 



Arizona 



California 



Connecticut 
Florida * 
Idaho 
Illinois 
Kansas „ 
Louisiana 
Maine 
Michigan 

Minnesota 
Montana 



DISTRICT 
Jackson County 

Anchorage Area Greater Borough, Gateway 
Borough, Idltorod Area^ Kake City. 
Matsanuka/Susi tna Borough, Lower Yukon, 
Unalaska City ; 

Marana High School , Scottsdale, Page Unified, 
Peach Springs Elementary, Puerco Elementary 
#018, Sunnyslde Unified #12, Tuba City Uni- 
fied flSp Union Elementary #062, Whltiriver '* 
Unified #20 , W1 ndow. Rock El^ntar^ #008 ^ ■ : 

Clpvli Unified, Fal rfi el d«5u1 sun Public 
School, Fontana Unified, Huntington Batch 
Union High School , Oakland C Uy Unified, 
Qroville Union * San Juan Unified, San 
Jacinto Unified, Sacramento^ G1 ty Uni fled, 
San Pasqual Val 1 ey Uni fled/ Shasta Lake 
Union Elementary, Tracy Elementary 

Bridgeport 

Dade County ^ 
Blackfoot #65 ' 
Rockford 

Lawrence Public Schools 

La Fourehe Parish 

Maine Attn. #29 . V" 

Brlroley, Escanaba Area Public Schools, 
Grade Haven City, Marquette City, Mason 
Consolidated, Lake Shore, Wayne-Westl and 
Coranuhtiy School * - 

Rohblnsdale Independent #281, Frazee, Duluth 
Independent #709, Waubun Independent 1435 

Box Elder Elementary #55, Brockton Elementary 
#55, Hays/Lodge Pole, Lame Deer El erflenta ry, 
Hyola Elementary #29 * ,\ 



Nebraska 



Falls City Public School 



STATE ; 
Nevada 
New Jersey 
New Mexico r 

New York 

North Carolina 

' . i "- 

North Dakota 
Okl ahoma 



Oregon 
South Dakota 

Texas 
Utah 

Washingtdn 

Ml sconsin 
Wyoml ng 



DISTRICT . 

White Mfle County 

Mahwah Township 

Bernal 11 lov Grants Municipal Schools, 
Sail up-McKI nl ey County , Taos 

Gowanda, Buffalo - i 



Columbus County, Robeson County, Graham { 
County \ / . J . 

Couture #27, Mil 11 ston A 

Boswell Elementary #1, Broken Bow, CNcoiali 
1-19, Commerce, Edmond, Southslde, El fteflo , 
#3% Fairfax; Idibel .^^rtf*uiit^iW/^*^ 
MeAl ester 0-80, Norwood, Oklahoma City, l 
McCurtaln County, Pea vine #19, ArdBior'%, 
Okmulgee, Puree! 1 , Ryan, Geary, Tall Ulna* 
Trl -County Area AVTS. , Tuskahoma, Whitehead, ; 
zion #28 ..- ■■ - : V. •.' ■ i-'.:^ 

Klamath Coun^ Unified, Lincoln County, -H M 
Umatilla County Intermediate ' j 

Rapid City, Todd County, Sisseton, ' ] 

Lake Andes „ .;;} 

Dallas Intermediate 

San yuan, Tooele 

Bell Ingham #501, Goldendale, Hoqulam #28, 
Nespelem#14, Ronton D-403, Toutle Lake #11, 
Wapato #207 

LaCrosse Joint, Flambeau #1, Hayward 
Community * , J 

Fremont County #25 
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